An automatic object detection method for microscopic images based on attention mechanism

Hao Ruqian; Wang Xiangzhou; Zhang Jing; Liu Juanxiu; Du Xiaohui; Liu Lin

doi:10.12086/oee.2022.210361

Article navigation > Opto-Electronic Engineering > 2022 Vol. 49 > No. 3 > 210361

Next Article Previous Article

Hao R Q, Wang X Z, Zhang J, et al. An automatic object detection method for microscopic images based on attention mechanism[J]. Opto-Electron Eng, 2022, 49(3): 210361. doi: 10.12086/oee.2022.210361

Citation:

Hao R Q, Wang X Z, Zhang J, et al. An automatic object detection method for microscopic images based on attention mechanism[J]. Opto-Electron Eng, 2022, 49(3): 210361. doi: 10.12086/oee.2022.210361

An automatic object detection method for microscopic images based on attention mechanism

School of Optoelectronic Science and Engineering, University of Electronic Science and Technology of China, Chengdu, Sichuan 611731, China

Fund Project: National Natural Science Foundation of China (61905036) and the Fundamental Research Funds for the Central Universities (ZYGX2021YGCX020)

More Information

^*Corresponding author: juanxiul@qq.com

Received Date 13 November 2021

Revised Date 11 January 2022

Published Date 25 March 2022

Abstract

Abstract

The microscopic image has the characteristics of complex background and overlapping cells. Due to the technical limitations, traditional image processing methods cannot accurately complete the real-time recognition task. To address the above-mentioned problems, we propose an automatic detection method for microscopic images using attention mechanism. This method improves the original DETR architecture by introducing a split-transform-merge mechanism, which reduces the dimensionality of input features and trains multiple groups of convolution kernels for feature extraction, thereby effectively improving the model's feature extraction ability for the targets and increasing the accuracy of model detection rate. The experimental results show that the mAP of the improved model was 96.3%, which is 10% higher than that of the original model DETR. Meanwhile, the proposed method has superior detection capabilities for scenarios such as cell overlap, adhesion, and complex background. Moreover, the detection time for each leucorrhea image was about 88.8 ms, which can satisfy the requirement of real-time microscopy examination.
- microscopic image /
- attention mechanism /
- deep learning /
- vaginitis /
- automatic detection

FullText(HTML)

References

[1]	Paczkowska M, Chanaj-Kaczmarek J, Romaniuk-Drapała A, et al. Mucoadhesive chitosan delivery system with Chelidonii herba lyophilized extract as a promising strategy for vaginitis treatment[J]. J Clin Med, 2020, 9(4): 1208. doi: 10.3390/jcm9041208 CrossRef Google Scholar
[2]	Abdul-Aziz M, Mahdy M A K, Abdul-Ghani R, et al. Bacterial vaginosis, vulvovaginal candidiasis and trichomonal vaginitis among reproductive-aged women seeking primary healthcare in Sana'a city, Yemen[J]. BMC Infect Dis, 2019, 19(1): 879. doi: 10.1186/s12879-019-4549-3 CrossRef Google Scholar
[3]	Mylonas I, Bergauer F. Diagnosis of vaginal discharge by wet mount microscopy: a simple and underrated method[J]. Obstet Gynecol Surv, 2011, 66(6): 359−368. doi: 10.1097/OGX.0b013e31822bdf31 CrossRef Google Scholar
[4]	田鹏, 严伟, 李凡星, 等. 均匀球面波数字同轴全息生物显微方法[J]. 光电工程, 2019, 46(1): 180110. Google Scholar Tian P, Yan W, Li F X, et al. Biology microscopy using well-distributed sphere digital in-line holography[J]. Opto-Electron Eng, 2019, 46(1): 180110. Google Scholar
[5]	Zhang S L, Liu L W, Ren S, et al. Recent advances in nonlinear optics for bio-imaging applications[J]. Opto-Electron Adv, 2020, 3(10): 200003. doi: 10.29026/oea.2020.200003 CrossRef Google Scholar
[6]	顾鑫, 黄伟, 杨立梅, 等. 微流体衍射相位显微成像及其在寄生虫测量中的应用[J]. 光电工程, 2019, 46(12): 190046. Google Scholar Gu X, Huang W, Yang L M, et al. Microfluidic diffraction phase microscopy and its application in parasites measurement[J]. Opto-Electron Eng, 2019, 46(12): 190046. Google Scholar
[7]	崔光茫, 张克奇, 毛磊, 等. 结合多尺度分解和梯度绝对值算子的显微图像清晰度评价方法[J]. 光电工程, 2019, 46(6): 180531. Google Scholar Cui G M, Zhang K Q, Mao L, et al. Micro-image definition evaluation using multi-scale decomposition and gradient absolute value[J]. Opto-Electron Eng, 2019, 46(6): 180531. Google Scholar
[8]	杜晓辉, 刘霖, 张静, 等. 基于LBP纹理特征的白带显微图像中上皮细胞检测方法[J]. 液晶与显示, 2019, 34(9): 871−878. Google Scholar Du X H, Liu L, Zhang J, et al. Detection of epithelial cells in leucorrhea microscopic images based on LBP texture features[J]. Chin J Liquid Crys Dis, 2019, 34(9): 871−878. Google Scholar
[9]	Krizhevsky A, Sutskever I, Hinton G E. ImageNet classification with deep convolutional neural networks[C]//Proceedings of the 25th International Conference on Neural Information Processing Systems, 2012. Google Scholar
[10]	Wen T X, Wu H X, Du Y, et al. Faster R-CNN with improved anchor box for cell recognition[J]. Math Biosci Eng, 2020, 17(6): 7772−7786. doi: 10.3934/mbe.2020395 CrossRef Google Scholar
[11]	Jia D Y, Zhou J L, Zhang C W. Detection of cervical cells based on improved SSD network[J]. Multimed Tools Appl (2021). doi: 10.1007s11042-021-11015-7. Google Scholar
[12]	Jiang Z F, Liu X, Yan Z Z, et al. Improved detection performance in blood cell count by an attention-guided deep learning method[J]. OSA Continuum, 2021, 4(2): 323−333. doi: 10.1364/OSAC.413787 CrossRef Google Scholar
[13]	Carion N, Massa F, Synnaeve G, et al. 2020. End-to-end object detection with transformers[C]//Proceedings of the 16th European Conference on Computer Science, 2020: 213–229. Google Scholar
[14]	Prangemeier T, Reich C, Koeppl H. Attention-based transformers for instance segmentation of cells in microstructures[C]//Proceedings of 2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), 2020: 700–707. Google Scholar
[15]	He K M, Zhang X Y, Ren S Q, et al. Deep residual learning for image recognition[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016: 770–778. Google Scholar
[16]	Kuhn H W. The Hungarian method for the assignment problem[J]. Nav Res Logist Quart, 1955, 2(1-2): 83−97. doi: 10.1002/nav.3800020109 CrossRef Google Scholar
[17]	Xie S N, Girshick R, Dollár P, et al. Aggregated residual transformations for deep neural networks[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition, 2017: 5987–5995. Google Scholar

Overview

Overview

The microscopic image has the characteristics of complex background and overlapping cells. The traditional microscopy examination is performed by experienced doctors, and usually takes long time. Vaginitis is a common and potentially harmful gynecological disease. It can significantly increase the risk of women being infected with pelvic inflammatory disease, Human Immunodeficiency Virus (HIV), and premature birth. Routine leucorrhea examination is a common early screening method for vaginitis, but traditional microscopic examination takes a long time, and the test results are largely affected by the subjectivity of the doctor. Due to the technical limitations, traditional image processing methods cannot accurately complete the real-time recognition task. To address the above-mentioned problems, we propose an automatic object detection method for microscopic images based on attention mechanism and perform it on the scenario of vaginitis screen. This method can achieve real-time and efficient detection of three common cells, which are mildews, trichomonas and clue cells. The proposed framework uses DETR architecture based on the self-attention mechanism for detection. Different from the commonly used object detection deep learning models, DETR employs Hungarian bipartite match algorithm to assign the prediction and ground truth based on the least cost, which saves the procedure of generating anchor boxes and region of interest proposal. The self-attention mechanism is also used in DETR to generate the global connection of the targets and the whole image, which enables the model to detect the overlapped targets more accurately. In our scenario of microscopic images with complex background, to achieve better performance in image feature extraction section, we replaced the original CNN backbone with a CNN architecture with a split-transform-merge mechanism. It reduces the dimensionality of input features and trains multiple groups of convolution kernels for feature extraction, thereby effectively improving the model's feature extraction ability for the targets and increasing the accuracy of model detection rate. The experimental results show that the mAP of the improved model was 96.3%, which is 10% higher than that of the original model DETR. Meanwhile, the proposed method has superior detection capabilities for scenarios such as cell overlap, adhesion, and complex background. Moreover, the detection time for each leucorrhea image was about 88.8 ms, which can satisfy the requirement of real-time detection. Therefore, this model can accurately identify pathogenic microorganisms in real time and meet the needs of actual clinical use for leucorrhea routine examination. In addition, this method has high transferability and robustness, which can be quickly migrated to microscopic clinical applications such as routine fecal microscopy, urine composition analysis, blood routine analysis, etc.