Cylindrical image mosaic method based on fast camera calibration in multi-scene

Fu Ziqiu; Zhang Xiaolong; Yu Chen; Liang Dan; Liang Dongtai

doi:10.12086/oee.2020.190436

Article navigation > Opto-Electronic Engineering > 2020 Vol. 47 > No. 4 > 190436

Next Article Previous Article

Fu Z Q, Zhang X L, Yu C, et al. Cylindrical image mosaic method based on fast camera calibration in multi-scene[J]. Opto-Electron Eng, 2020, 47(4): 190436. doi: 10.12086/oee.2020.190436

Citation:

Fu Z Q, Zhang X L, Yu C, et al. Cylindrical image mosaic method based on fast camera calibration in multi-scene[J]. Opto-Electron Eng, 2020, 47(4): 190436. doi: 10.12086/oee.2020.190436

Cylindrical image mosaic method based on fast camera calibration in multi-scene

1.
Faculty of Mechanical Engineering and Mechanics, Ningbo University, Ningbo, Zhejiang 315211, China
2.
Shanghai Pujiang Bridge and Tunnel Operation Management Co., Ltd, Shanghai 200023, China
3.
Ningbo Hermeneutic Aircraft Machinery Technology Co., Ltd, Ningbo, Zhejiang 315200, China

Fund Project: Supported by National Natural Science Foundation of China (51805280), the Public Technology Application Project of Zhejiang (2017C31094), and Natural Science Foundation of Zhejiang (LQ18E050005)

More Information

Corresponding author: Liang Dongtai, E-mail:liangdongtai@nbu.edu.cn

Received Date 24 July 2019

Revised Date 29 November 2019

Published Date 01 April 2020

Abstract

Abstract

A cylindrical image mosaic method based on fast camera calibration in multi-scene is proposed to solve the problems of scene limitation and complex calibration process in image mosaic using camera calibration parameter. Firstly, the accurate corner feature of checkerboard calibration board is used to make it in the overlapping field of view of two adjacent images. Then, the image sequence is pre-processed by corner extraction, precision and matching, so that the registration parameters between the images to be stitched can be solved accurately and quickly. After that, the cylindrical projection is used to maintain the visual consistency of the images, and the multi-band fusion is used to retain the details of the images. Subsequently, the images are stitched using registration parameters obtained by calibration. Finally, the whole system is built on a low-power embedded platform to accomplish fast calibration and mosaic process based on calibration parameters in multi-scene. The experiment results show that the proposed method can accomplish camera calibration quickly and accurately in indoor and tunnel scenarios, and the image mosaic process is time-consuming. Meanwhile, it can ensure better stitching accuracy and imaging effect, and has strong robustness.
- image mosaic /
- camera calibration /
- cylindrical projection /
- multi-band fusion

FullText(HTML)

References

[1]	Adel E, Elmogy M, Elbakry H. Image stitching based on feature extraction techniques: a survey[J]. International Journal of Computer Applications, 2014, 99(6): 1-8. doi: 10.5120/17374-7818 CrossRef Google Scholar
[2]	Cheng Y F, Jin S Y, Wang M, et al. Image mosaicking approach for a double-camera system in the GaoFen2 optical remote sensing satellite based on the big virtual camera[J]. Sensors, 2017, 17(6): E1441. doi: 10.3390/s17061441 CrossRef Google Scholar
[3]	Bosch J, Gracias N, Ridao P, et al. Omnidirectional underwater camera design and calibration[J]. Sensors, 2015, 15(3): 6033-6065. doi: 10.3390/s150306033 CrossRef Google Scholar
[4]	陈晨涛, 潘之玮, 沈会良, 等.一种人体热红外图像拼接及部位划分方法[J].光电工程, 2019, 46(9): 180689. doi: 10.12086/oee.2019.180689 CrossRef Google Scholar Chen C T, Pan Z W, Shen H L, et al. Image stitching and partitioning algorithms for infrared thermal human-body images[J]. Opto-Electronic Engineering, 2019, 46(9): 180689. doi: 10.12086/oee.2019.180689 CrossRef Google Scholar
[5]	Kaynig V, Fischer B, Müller E, et al. Fully automatic stitching and distortion correction of transmission electron microscope images[J]. Journal of Structural Biology, 2010, 171(2): 163-173. doi: 10.1016/j.jsb.2010.04.012 CrossRef Google Scholar
[6]	吴丽萍, 胡郁.柱面全景图图像拼接中图像平滑的虚拟现实技术[J].科学技术与工程, 2017, 17(31): 271-276. doi: 10.3969/j.issn.1671-1815.2017.31.044 CrossRef Google Scholar Wu L P, Hu Y. Virtual reality technology of image smoothing in cylindrical panoramic image mosaic[J]. Science Technology and Engineering, 2017, 17(31): 271-276. doi: 10.3969/j.issn.1671-1815.2017.31.044 CrossRef Google Scholar
[7]	Seo S, Jeonz S, Lee S. Efficient homography estimation method for panorama[C]//Proceedings of the 19th Korea-Japan Joint Workshop on Frontiers of Computer Vision, 2013: 209-212. Google Scholar
[8]	Szeliski R. Video mosaics for virtual environments[J]. IEEE Computer Graphics and Applications, 1996, 16(2): 22-30. doi: 10.1109/38.486677 CrossRef Google Scholar
[9]	Zou L H, Chen J, Zhang J, et al. An image mosaicing approach for video sequences based on space-time manifolds[C]//Proceedings of the 29th Chinese Control Conference, 2010. Google Scholar
[10]	Li L N, Geng N. Algorithm for sequence image automatic mosaic based on sift feature[C]//Proceedings of 2010 Wase International Conference on Information Engineering, 2010. Google Scholar
[11]	Wu Z W, Zhu L R, Sun X C. Expansion of the visual angle of a car rear-view image via an image mosaic algorithm[J]. Optical Engineering, 2015, 54(5): 053101. doi: 10.1117/1.OE.54.5.053101 CrossRef Google Scholar
[12]	Zhou A W, Shao W, Guo J J. An image mosaic method for defect inspection of steel rotary parts[J]. Journal of Nondestructive Evaluation, 2016, 35(4): 60. doi: 10.1007/s10921-016-0375-3 CrossRef Google Scholar
[13]	Alomran M, Chai D. Feature-based panoramic image stitching[C]//Proceedings of 2016 14th International Conference on Control, 2016. Google Scholar
[14]	张金玲, 孙汉旭, 贾庆轩, 等.基于相机标定的图像拼接算法[J].中北大学学报(自然科学版), 2008, 29(6): 575-579. doi: 10.3969/j.issn.1673-3193.2008.06.021 CrossRef Google Scholar Zhang J L, Sun H X, Jia Q X, et al. Image mosaic algorithm based on camera calibration[J]. Journal of North University of China (Natural Science Edition), 2008, 29(6): 575-579. doi: 10.3969/j.issn.1673-3193.2008.06.021 CrossRef Google Scholar
[15]	王冬, 刘凤英, 陈天恩, 等.一种相机标定参数的柱面全景影像拼接方法[J].测绘科学, 2016, 41(7): 150-154, 143. Google Scholar Wang D, Liu F Y, Chen T E, et al. The method of cylinder panoramic image mosaic based on camera calibration parameters[J]. Science of Surveying and Mapping, 2016, 41(7): 150-154, 143. Google Scholar
[16]	马嘉琳, 张锦明, 孙卫新.基于相机标定的全景图拼接方法研究[J].系统仿真学报, 2017, 29(5): 1112-1119. Google Scholar Ma J L, Zhang J M, Sun W X. Research on panoramic image mosaic method based on camera calibration[J]. Journal of System Simulation, 2017, 29(5): 1112-1119. Google Scholar
[17]	Zhang Z Y. Flexible camera calibration by viewing a plane from unknown orientations[C]//Proceedings of the Seventh IEEE International Conference on Computer Vision, 1999. Google Scholar
[18]	Hartley R, Zisserman A. Multiple view geometry in computer vision[M]. Cambridge university press, 2003 Google Scholar
[19]	Chuan Z, Long T D, Feng Z, et al. A planar homography estimation method for camera calibration[C]//Proceedings 2003 IEEE International Symposium on Computational Intelligence in Robotics and Automation. Computational Intelligence in Robotics and Automation for the New Millennium, 2003. Google Scholar
[20]	Lenth R V. Least-squares means: the R package lsmeans[J]. Journal of Statistical Software, 2016, 69(1): 1-33. Google Scholar
[21]	Janota A, Šimák V, Nemec D, et al. Improving the precision and speed of euler angles computation from low-cost rotation sensor data[J]. Sensors, 2015, 15(3): 7016-7039. doi: 10.3390/s150307016 CrossRef Google Scholar
[22]	Pulli K, Baksheev A, Kornyakov K, et al. Real-time computer vision with OpenCV[J]. Communications of the ACM, 2012, 55(6): 61-69. doi: 10.1145/2184319.2184337 CrossRef Google Scholar
[23]	Xu Y, Zhou Q H, Gong L W, et al. High-speed simultaneous image distortion correction transformations for a multicamera cylindrical panorama real-time video system using FPGA[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2014, 24(6): 1061-1069. doi: 10.1109/TCSVT.2013.2290576 CrossRef Google Scholar
[24]	郭海霞, 郭海龙, 解凯.基于边缘信息改进的双线性插值算法[J].计算机工程与应用, 2011, 47(31): 171-174. doi: 10.3778/j.issn.1002-8331.2011.31.049 CrossRef Google Scholar Guo H X, Guo H L, Xie K. Improved bilinear interpolation algorithm based on edge information[J]. Computer Engineering and Applications, 2011, 47(31): 171-174. doi: 10.3778/j.issn.1002-8331.2011.31.049 CrossRef Google Scholar
[25]	郭治成, 党建武, 王阳萍, 等.基于多特征融合的背景建模方法[J].光电工程, 2018, 45(12): 180206. doi: 10.12086/oee.2018.180206 CrossRef Google Scholar Guo Z C, Dang J W, Wang Y P, et al. Background modeling method based on multi-feature fusion[J]. Opto-Electronic Engineering, 2018, 45(12): 180206. doi: 10.12086/oee.2018.180206 CrossRef Google Scholar
[26]	Brown M, Lowe D G. Automatic panoramic image stitching using invariant features[J]. International Journal of Computer Vision, 2007, 74(1): 59-73. Google Scholar

Overview

Overview

Overview: Image mosaic is the process of combining two or more images with an overlapping field of view in the same scene to produce a seamless panorama or high-resolution image. The image obtained by mosaic has a larger field of view (FOV). Most of the cameras have a FOV angle of about 35 × 50 degrees, which limits the acquisition of information. Therefore, through image mosaic, the continuous image sequence of the same scene is stitched to form a composite image with a larger FOV, which can obtain all visual information at a given point of view at one time. This technology plays an important role in many fields, such as geological survey, medical minimally invasive surgery, and virtual reality. Its technical advantages are obvious. Researchers at home and abroad have done a lot of researches on image mosaic, and image registration is the key step. At present, there are mainly three classical registration methods based on frequency domain, gray level, and feature points, respectively. Nevertheless, the above image registration algorithms generally have the problems of large computational load and low execution efficiency. In this regard, some domestic researchers have proposed a camera calibration method, which saves most of the time needed for stitching and achieves high stitching accuracy.

But at present, the mosaic algorithm based on camera calibration is limited by the scene, and the calibration process is complex. The collinear condition in imaging will be destroyed after image transformation, which is not conducive to subsequent image processing and information classification. For this reason, a cylindrical image mosaic method based on fast camera calibration in multi-scene is proposed. This method makes full use of the high accuracy of feature extraction of checkerboard calibration board, which is used to make it in the overlapping field of view of two adjacent images. In order to accurately and quickly solve the registration parameters between the images to be spliced, the image sequence is pre-processed by corner extraction, precision and matching. Then, cylindrical projection and multi-band fusion are used to maintain visual consistency and detailed information. The system is based on a Low-Power Embedded platform, which achieves fast acquisition and accurate mosaic of camera calibration parameters in multi-scene. The experiment results show that the proposed method can accomplish camera calibration quickly and accurately in indoor and tunnel scenarios, and the image mosaic process is time-consuming. Meanwhile, it can ensure better stitching accuracy and imaging effect, and has strong robustness.

The proposed method has positive significance for real-time image stitching without feature points or large environmental changes.