Satellite cloud image retrieval based on deep metric learning

Jin Zhuzhang; Fang Xuyuan; Huang Yanhui; Yin Caoqian; Jin Wei

doi:10.12086/oee.2022.210307

Article navigation > Opto-Electronic Engineering > 2022 Vol. 49 > No. 4 > 210307

Next Article Previous Article

Jin Z Z, Fang X Y, Huang Y H, et al. Satellite cloud image retrieval based on deep metric learning[J]. Opto-Electron Eng, 2022, 49(4): 210307. doi: 10.12086/oee.2022.210307

Citation:

Jin Z Z, Fang X Y, Huang Y H, et al. Satellite cloud image retrieval based on deep metric learning[J]. Opto-Electron Eng, 2022, 49(4): 210307. doi: 10.12086/oee.2022.210307

Satellite cloud image retrieval based on deep metric learning

Faculty of Electrical Engineering and Computer Science, Ningbo University, Ningbo, Zhejiang 315211, China

Fund Project: National Natural Science Foundation of China (42071323) and Public Welfare Science and Technology Project of Ningbo (202002N3104).

More Information

Corresponding author: xyjw1969@126.com

Received Date 23 September 2021

Revised Date 18 January 2022

Available Online 20 April 2022

Published Date 25 April 2022

Abstract

Abstract

Due to the traditional cloud image retrieval methods are difficult to obtain ideal retrieval accuracy and retrieval efficiency, a cloud image retrieval method based on deep metric learning is proposed. Firstly, a residual 3D-2D convolutional neural network is designed to extract spatial and spectral features of cloud images. Since the features extracted by the traditional classify-based deep network may have greater differences intra-classes than inter-classes, the triplet strategy is used to train the network, and the cloud images are mapped into the metric space according to the similarity between cloud images, so that the distance of similar cloud images in the embedded space is smaller than that of non-similar cloud images. In model training, the convergence performance of traditional triplet loss is improved and the precision of cloud image retrieval is increased by adding a constraint on the distance between positive sample pairs to the lossless triplet loss function. Finally, through hash learning, the cloud features in the metric space are transformed into hash codes, so as to ensure the retrieval accuracy and improve the retrieval efficiency. Experimental results show that the mean average precision (mAP) of the proposed algorithm is 75.14% and 80.14% for the southeast coastal cloud image dataset and the northern hemisphere cloud image dataset respectively, which is superior to other comparison methods.
- deep learning /
- metric learning /
- triplet loss /
- satellite cloud image retrieval

FullText(HTML)

References

[1]	徐坚. 基于内容的卫星云图检索系统设计与实现[D]. 南京: 河海大学, 2004. Google Scholar Xu J. Contented based cloudy satellite images retrieval system design and practice[D]. Nanjing: Hohai University, 2004. Google Scholar
[2]	上官伟. 基于内容的卫星云图处理与信息检索技术研究[D]. 哈尔滨: 哈尔滨工程大学, 2008. Google Scholar Shangguan W. Research of content-based satellite cloud image processing and information retrieval technology[D]. Harbin: Harbin Engineering University, 2008. Google Scholar
[3]	李秀馨. 基于内容的卫星云图检索技术研究[D]. 南京: 南京航空航天大学, 2013. Google Scholar Li X X. Research on the technology of content-based satellite cloud image retrieval[D]. Nanjing: Nanjing University of Aeronautics and Astronautics, 2013. Google Scholar
[4]	徐晔烨. 基于多特征的红外云图检索技术研究[D]. 南京: 南京航空航天大学, 2014. Google Scholar Xu Y Y. Infrared cloud image retrieval technology research based on multi-feature[D]. Nanjing: Nanjing University of Aeronautics and Astronautics, 2014. Google Scholar
[5]	Zhang C J, Li G D, Lei R M, et al. Deep feature aggregation network for hyperspectral remote sensing image classification[J]. IEEE J Select Top Appl Earth Observ Remote Sens, 2020, 13: 5314−5325. doi: 10.1109/JSTARS.2020.3020733 CrossRef Google Scholar
[6]	Fang J S, Fu H Z, Liu J. Deep triplet hashing network for case-based medical image retrieval[J]. Med Image Anal, 2021, 69: 101981. doi: 10.1016/j.media.2021.101981 CrossRef Google Scholar
[7]	Ronneberger O, Fischer P, Brox T. U-Net: convolutional networks for biomedical image segmentation[C]//Proceedings of the 2015 18th International Conference on Medical Image Computing and Computer-Assisted Intervention, 2015: 234–241. Google Scholar
[8]	Roy S K, Krishna G, Dubey S R, et al. HybridSN: exploring 3-D–2-D CNN feature hierarchy for hyperspectral image classification[J]. IEEE Geosci Remote Sens Lett, 2020, 17(2): 277−281. doi: 10.1109/LGRS.2019.2918719 CrossRef Google Scholar
[9]	Feng F, Wang S T, Wang C Y, et al. Learning deep hierarchical spatial–spectral features for hyperspectral image classification based on residual 3D-2D CNN[J]. Sensors, 2019, 19(23): 5276. doi: 10.3390/s19235276 CrossRef Google Scholar
[10]	Hermans A, Beyer L, Leibe B. In defense of the triplet loss for person re-identification[Z]. arXiv: 1703.07737, 2017. https://doi.org/10.48550/arXiv.1703.07737 Google Scholar
[11]	Kaya M, Bilge H Ş. Deep metric learning: A survey[J]. Symmetry, 2019, 11(9): 1066. doi: 10.3390/sym11091066 CrossRef Google Scholar
[12]	Schroff F, Kalenichenko D, Philbin J. FaceNet: a unified embedding for face recognition and clustering[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition, 2015: 815–823. Google Scholar
[13]	Bessho K, Date K, Hayashi M, et al. An introduction to Himawari-8/9—Japan’s new-generation geostationary meteorological satellites[J]. J Meteor Soc Jpn Ser II, 2016, 94(2): 151−183. doi: 10.2151/jmsj.2016-009 CrossRef Google Scholar
[14]	郁凡, 陈渭民. 双光谱云图的云分类探讨[J]. 南京气象学院学报, 1994, 17(1): 117−124. Google Scholar Yu F, Chen W M. Research on the cloud classification for the bi-spectrum cloud picture[J]. J Nanjing Inst Meteor, 1994, 17(1): 117−124. Google Scholar
[15]	Bai C, Zhang M J, Zhang J L, et al. LSCIDMR: large-scale satellite cloud image database for meteorological research[J]. IEEE Trans Cybern, 2021. doi: 10.1109/TCYB.2021.3080121 CrossRef Google Scholar
[16]	He K M, Zhang X Y, Ren S Q, et al. Deep residual learning for image recognition[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016: 770–778. Google Scholar
[17]	Huang K K, Ren C X, Liu H, et al. Hyperspectral image classification via discriminative convolutional neural network with an improved triplet loss[J]. Pattern Recogn, 2021, 112: 107744. doi: 10.1016/j.patcog.2020.107744 CrossRef Google Scholar
[18]	Roy S, Sangineto E, Demir B, et al. Metric-learning-based deep hashing network for content-based retrieval of remote sensing images[J]. IEEE Geosci Remote Sens Lett, 2021, 18(2): 226−230. doi: 10.1109/LGRS.2020.2974629 CrossRef Google Scholar
[19]	Liu H M, Wang R P, Shan S G, et al. Deep supervised hashing for fast image retrieval[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016: 2064–2072. Google Scholar
[20]	Kim S, Seo M, Laptev I, et al. Deep metric learning beyond binary supervision[C]//Proceedings of the 2017 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 2283–2292. Google Scholar
[21]	Arsenault Marc-Olivier. Lossless Triplet loss[EB/OL]. (2018-02-15). http://coffeeanddata.ca/lossless-triplet-loss. Google Scholar
[22]	Cheng D, Gong Y H, Zhou S P, et al. Person Re-identification by multi-channel parts-based CNN with improved triplet loss function[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016: 1335–1344. Google Scholar
[23]	Xuan H, Stylianou A, Pless R. Improved embeddings with easy positive triplet mining[C]//Proceedings of the 2020 IEEE Winter Conference on Applications of Computer Vision, 2020: 2463–2471. Google Scholar
[24]	廖列法, 李志明, 张赛赛. 基于深度残差网络的迭代量化哈希图像检索方法[J]. 计算机应用, 2021. doi: 10.11772/j.issn.1001-9081.2021071135 CrossRef Google Scholar Liao L F, Li Z M, Zhang S S. Image retrieval method based on deep residual network and iterative quantization hashing[J]. J Comput Appl, 2021. doi: 10.11772/j.issn.1001-9081.2021071135 CrossRef Google Scholar
[25]	Liu W, Wang J, Ji R R, et al. Supervised hashing with kernels[C]//Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, 2012: 2074–2081. Google Scholar
[26]	Lin K, Yang H F, Hsiao J H, et al. Deep learning of binary hash codes for fast image retrieval[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2015: 27–35. Google Scholar
[27]	Torralba A, Murphy K P, Freeman W T, et al. Context-based vision system for place and object recognition[C]//Proceedings of the Ninth IEEE International Conference on Computer Vision, 2003: 273–280. Google Scholar

Overview

Overview

Meteorological satellites can monitor weather phenomena of different scales from the air, and the satellite cloud images obtained by them play an important role in weather analysis and forecast. In recent years, with the development of meteorological satellite technology, the spatial and spectral resolution of satellite cloud images and the acquisition frequency of imaging spectrometer have been continuously improved. How to manage massive satellite cloud images and design an efficient cloud image retrieval system has become a difficult problem for meteorologists. However, the traditional cloud image retrieval methods are difficult to obtain ideal retrieval accuracy and retrieval efficiency. Motivated by the impressive success of the modern deep neural network (DNN) in learning the optimization features of specific tasks in an end-to-end fashion, a cloud image retrieval method based on deep metric learning is proposed in this paper. Firstly, a residual 3D-2D convolutional neural network was designed to extract spatial and spectral features of cloud images. Since the features extracted by the traditional classify-based deep network may have greater differences intra-classes than inter-classes, the triplet strategy is used to train the network, and the cloud images are mapped into the metric space according to the similarity between cloud images, so that the distance of similar cloud images in the embedded space is smaller than that of non-similar cloud images. In model training, the convergence performance of traditional triplet loss is improved and the precision of cloud image retrieval is increased by adding a constraint on the distance between positive sample pairs to the lossless triplet loss function. Finally, through hash learning, the cloud features in the metric space are transformed into hash codes, so as to ensure the retrieval accuracy and improve the retrieval efficiency. Experimental results show that the mean average precision (mAP) of the proposed algorithm is 75.14% and 80.14% for the southeast coastal cloud image dataset and the northern hemisphere cloud image dataset respectively, which is superior to other comparison methods.