Open Access Open Access  Restricted Access Subscription Access


Van The Thanh, Do Quang Khoi, Le Huu Ha, Le Manh Thanh


The problem of finding and identifying semantics of images is applied in multimedia applications of many different fields such as Hospital Information System, Geographic Information System, Digital Library System, etc. In this paper, we propose the semantic-based image retrieval (SBIR) system based on the deep learning technique; this system is called as SIR-DL that generates visual semantics based on classifying image contents. At the same time we identify the semantics of similar images on Ontology, which describes semantics of visual features of images. Firstly, the color and spatial features of segmented images are we extracted and these visual feature vectors are trained on the deep neural network to obtain visual words vectors. The process of image retrieval is executed rely on semantic classification of SIR-DL according to the visual feature vector of the query image from which it produces a visual word vector. Then, we retrieve it on Ontology to provide the identities and the semantics of similar images corresponds to a similarity measure. In order to carry out SIR-DL, the algorithms and diagram of this image retrieval system are proposed after that we implement them on ImageCLEF@IAPR, which has 20,000 images. On the base of the experimental results, the effectiveness of our method is evaluated by the accuracy, precision, recall, and F-measure; these results are compared with some of works recently published on the same image dataset. It shows that SIR-DL effectively solves the problem of semantic-based image retrieval and can be used to build multimedia systems in many different fields.


bag of visual word; deep learning; ontology; SBIR; similarity measure; similar images

Full Text:



N. R. A. Alzubi, A. Amira, "Semantic content-based image retrieval: A comprehensive study", Journal of Visual Communication and Image Representation, vol. 32, no. xx, pp. 20-54, 2017.

L. J. A.B. Spanier, D. Cohen, "A new method for the automatic retrieval of medical cases based on the radlex ontology", International Journal of Computer Assisted Radiology and Surgery, vol. 12, no. 3, pp. 471-484, 2017.

Y. Alqasrawi, "Bridging the gap between local semantic concepts and bag of visual words for natural scene image retrieval", International Journal of Sensors Wireless Communications and Control, vol. 6, no. 3, pp. 174-191, 2016.

BusinessInsider. (2018). [Online].

L. S. C. Hernndez-Gracidas, "Markov random elds and spatial information to improve automatic image annotation", Advances in Image and Video Technology: Springer, Berlin, Heidelberg, 2007, pp. 879-892. [Online]. Available:

-3-540-77129-6 74

C. B. C. Kurtz, A. Depeursinge, "A semantic framework for the retrieval of similar radiological images based on medical annotations", IEEE International Conference on Image Processing, Paris, France: IEEE, 2004, pp. xx-xx. [Online]. Available:

M. M.-y.-G. C.A. Hernndez-Gracidas, L.E. Sucar, "Improving image retrieval by using spatial relations", Multimedia Tools and Applications, vol. 62, no. 2, pp. 479-505, 2013.

P. Carbonetto. (2018). [Online]. Available:

Deloitte, "Photo sharing: trillions and rising", Deloitte Touche Tohmatsu Limited, Deloitte Global, Tech. Rep., 2016.

Deloitte(b). (2018). [Online]. Available:

B. Z. Y. et al., "I2t: Image parsing to text description", Proceedings of the IEEE: IEEE, 2010, pp. 1485-1508. [Online]. Available:

C. B. et al., "Emerging semantic-based applications", Semantic Web, Cham: Springer, 2016, pp. 39-83. [Online]. Available: 4

H. E. et al., "The segmented and annotated iapr tc-12 benchmark", Computer Vision and Image Understanding, vol. In press, no. xx, pp. xx-xx, 2009.

S. F. et al., "Image retrieval based on using hamming distance" Procedia Comp. Sci., vol. 73, no. xx, pp. 320-327, 2015.

Z. Z. V. S. G. Castanon, Y. Chen, "Efficient activity retrieval through semantic graph queries", International conference on Multimedia, Brisbane, Australia: ACM, 2015, pp. 391-400.[Online]. Available:

M. Grubinger, "Analysis and evaluation of visual information systems performance", School of Computer Science and Mathematics, Faculty of Health, Engineering and Science, Victoria University, Melbourne, Australia, Tech. Rep., 2007.

S. O. H. Cevikalp, M. Elmas, "Large-scale image retrieval using transductive support vector machines", Computer Vision and Image Understanding, vol. xx, no. xx, pp. 1-11, 2017.

D. R. J. Gantz, "The digital universe in 2020: Big data, bigger digital shadows, and biggest growth in the far east", IDC iView, EMC Corporation, Tech. Rep., 2012 and 2014.

S. H. P. W.-J. Z. Y. Z. J. L. J. Wan, D. Wang, "Deep learning for content-based image retrieval: A comprehensive study", International conference on Multimedia, Orlando, Florida, USA: ACM, 2014, pp. 157-166. [Online]. Available:

H. A. L. Deligiannidis, "Emerging Trends in Image Processing: Computer Vision", and Pattern Recognition, Graduate Texts in Mathematics. Elsevier, USA: Morgan Kaufmann, Waltham,

MA 02451, 2015.

Y. Li, "Semantic image similarity based on deep knowledge for eective image retrieval", Department of Computer Science., Hong Kong Baptist University, Tech. Rep., 2014.

E.-I. C. M.-H. Lee, S. Rho, "Ontology based user query interpretation for semantic multimedia contents retrieval", Multimedia Tools and Applications, vol. 73, no. 2, pp. 901-915, 2014.

H. S. M. Jiu, "Nonlinear deep kernel learning for image annotation", IEEE Trans. on Image Processing, vol. 26, no. 4, pp. 1820-1832, 2017.

A. T. M. Tzelepi, "Deep convolutional learning for content based image retrieval", Neurocomputing, vol. 275, no. xx, pp. 2467-2478, 2018.

L. G. P. Muneesawang, N. Zhang, "Multimedia Database Retrieval: Technology and Applications", Graduate Texts in Mathematics. Springer, New York Dordrecht London, 2014.

T. M.-T. S.-A. R. M. M. S. Jabeen, Z. Mehmood, "An effective content-based image retrieval technique for image visuals representation based on the bag-of-visual-wordsmodel", PLoS ONE,

vol. 13, no. 4, pp. 1-24, 2018.

H. S. Pandey, PriteeKhanna, "A semantics and image retrieval system for hierarchical image databases", Information Processing Management, vol. 52, no. 4, pp. 571-591, 2016.

P. T.-M. L. V. Vijayarajan, M. Dinakaran, "A generic framework for ontology based information retrieval and image retrieval in web data", Human-centric Computing and Information Sciences, vol. 6, no. 18, pp. 1-30, 2016.

J. Z.-N. C.-Y. W. X. Xie, X. Cai, "A semantic-based method for visualizing large image collections", IEEE Transactions on Visualization and Computer Graphics, IEEE Computer Society, vol. xx, no. xx, pp. 1-15, 2018.

T. X.-C. X.-K. Y. W.-Y. M. T. Z. Y. Bai, W. Yu, "Bag-of-words based deep neural network for image retrieval", International conference on Multimedia, Orlando, Florida, USA: ACM, 2014, pp. 229-232. [Online]. Available:

J. W.-Q. Y.-P. Y. Y. Cao, M. Long, "Deep visual-semantic hashing for cross-modal retrieval", Inter. Conf. on Knowl. Discovery and Data Mining, California, USA: ACM, 2016, pp. 1445-1454. [Online]. Available:

D. Z. Y. Chou, D.J. Lee, "Semantic-based brain mri image segmentation using convolutional neural network", Advances in Visual Computing, Cham: Springer, 2016, pp. 628-638. [Online].

Available: 56

W. H. Z. Zheng, H. Bu, "An approach to classify visual semantic based on visual encoding with the convolutional neural network", International Conference on Fuzzy Systems and Knowledge Discovery, Zhangjiajie, China: IEEE, 2015, pp. xx-xx. [Online]. Available:

DOI: Display counter: Abstract : 184 views. PDF : 84 views.

Journal of Computer Science and Cybernetics ISSN: 1813-9663

Published by Vietnam Academy of Science and Technology