光场相机作为新一代的成像设备，能够同时捕获光线的空间位置和入射角度，然而其记录的光场存在空间分辨率和角度分辨率之间的制约关系，尤其子孔径图像有限的空间分辨率在一定程度上限制了光场相机的应用场景。因此本文提出了一种融合多尺度特征的光场图像超分辨网络，以获取更高空间分辨率的光场子孔径图像。该基于深度学习的网络框架分为三大模块：多尺度特征提取模块、全局特征融合模块和上采样模块。网络首先通过多尺度特征提取模块学习4D光场中固有的结构特征，然后采用融合模块对多尺度特征进行融合与增强，最后使用上采样模块实现对光场的超分辨率。在合成光场数据集和真实光场数据集上的实验结果表明，该方法在视觉评估和评价指标上均优于现有算法。另外本文将超分辨后的光场图像用于深度估计，实验结果展示出光场图像空间超分辨率能够增强深度估计结果的准确性。
融合多尺度特征的光场图像超分辨率方法
作者单位信息
通信作者： kirinshi@sjtu.edu.cn
出版日期：2020年12月22日
摘要
参考文献
[1] Lippmann G. épreuves réversibles donnant la sensation du relief[J]. Journal de Physique Théorique et Appliquée, 1908, 7(1): 821?825.
[2] Adelson E H, Wang J Y A. Single lens stereo with a plenoptic camera[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1992, 14(2): 99?106.
[3] Ng R, Levoy M, Brédif M, et al. Light field photography with a hand-held plenoptic camera[R]. Stanford Tech Report CTSR 2005-02, 2005.
[4] Tan Z P, Johnson K, Clifford C, et al. Development of a modular, high-speed plenoptic-camera for 3D flow-measurement[J]. Optics Express, 2019, 27(9): 13400?13415.
[5] Fahringer T W, Lynch K P, Thurow B S. Volumetric particle image velocimetry with a single plenoptic camera[J]. Measurement Science and Technology, 2015, 26(11): 115201.
[6] Shi S X, Ding J F, New T H, et al. Volumetric calibration enhancements for single-camera light-field PIV[J]. Experiments in Fluids, 2019, 60(1): 21.
[7] Shi S X, Ding J F, New T H, et al. Light-field camera-based 3D volumetric particle image velocimetry with dense ray tracing reconstruction technique[J]. Experiments in Fluids, 2017, 58(7): 78.
[8] Shi S X, Wang J H, Ding J F, et al. Parametric study on light field volumetric particle image velocimetry[J]. Flow Measurement and Instrumentation, 2016, 49: 70?88.
[9] Sun J, Xu C L, Zhang B, et al. Three-dimensional temperature field measurement of flame using a single light field camera[J]. Optics Express, 2016, 24(2): 1118?1132.
[10] Shi S X, Xu S M, Zhao Z, et al. 3D surface pressure measurement with single light-field camera and pressure-sensitive paint[J]. Experiments in Fluids, 2018, 59(5): 79.
[11] Ding J F, Li H T, Ma H X, et al. A novel light field imaging based 3D geometry measurement technique for turbomachinery blades[J]. Measurement Science and Technology, 2019, 30(11): 115901.
[12] Cheng Z, Xiong Z W, Chen C, et al. Light field super-resolution: a benchmark[C]//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Long Beach, CA, 2019.
[13] Lim J, Ok H, Park B, et al. Improving the spatail resolution based on 4D light field data[C]//Proceedings of the 16th IEEE International Conference on Image Processing, Cairo, Egypt, 2009, 2: 1173?1176.
[14] Georgiev T, Chunev G, Lumsdaine A. Superresolution with the focused plenoptic camera[J]. Proceedings of SPIE, 2011, 7873: 78730X.
[15] Bishop T E, Favaro P. The light field camera: extended depth of field, aliasing, and superresolution[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 34(5): 972?986.
[16] Rossi M, Frossard P. Graph-based light field super-resolution[C]//Proceedings of the IEEE 19th International Workshop on Multimedia Signal Processing, Luton, UK, 2017: 1?6.
[17] Alain M, Smolic A. Light field super-resolution via LFBM5D sparse coding[C]//Proceedings of the 25th IEEE International Conference on Image Processing, Athens, Greece, 2018: 1?5.
[18] Egiazarian K, Katkovnik V. Single image super-resolution via BM3D sparse coding[C]//Proceedings of the 23rd European Signal Processing Conference, Nice, France, 2015: 2849?2853.
[19] Alain M, Smolic A. Light field denoising by sparse 5D transform domain collaborative filtering[C]//Proceedings of the IEEE 19th International Workshop on Multimedia Signal Processing, Luton, UK, 2017: 1?6.
[20] Yoon Y, Jeon H G, Yoo D, et al. Learning a deep convolutional network for light-field image super-resolution[C]//Proceedings of 2015 IEEE International Conference on Computer Vision Workshop, Santiago, Chile, 2015: 57?65.
[21] Wang Y L, Liu F, Zhang K B, et al. LFNet: a novel bidirectional recurrent convolutional neural network for light-field image super-resolution[J]. IEEE Transactions on Image Processing, 2018, 27(9): 4274?4286.
[22] Zhang S, Lin Y F, Sheng H. Residual networks for light field image super-resolution[C]//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 2019: 11046?11055.
[23] Wang L G, Wang Y Q, Liang Z F, et al. Learning parallax attention for stereo image super-resolution[C]//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 2019: 12250?12259.
[24] Chen L C, Zhu Y K, Papandreou G, et al. Encoder-decoder with atrous separable convolution for semantic image segmentation[C]//Proceedings of the European Conference on Computer Vision, Glasgow, United Kingdom, 2018: 801?818.
[25] Wang R G, Liu L L, Yang J, et al. Image super-resolution based on clustering and collaborative representation[J]. Opto-Electronic Engineering, 2018, 45(4): 170537.
汪荣贵, 刘雷雷, 杨娟, 等. 基于聚类和协同表示的超分辨率重建[J]. 光电工程, 2018, 45(4): 170537.
[26] Shi W Z, Caballero J, Huszár F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, 2016: 1874?1883.
[27] Xu L, Fu R D, Jin W, et al. Image super-resolution reconstruction based on multi-scale feature loss function[J]. Opto-Electronic Engineering, 2019, 46(11): 180419.
徐亮, 符冉迪, 金炜, 等. 基于多尺度特征损失函数的图像超分辨率重建[J]. 光电工程, 2019, 46(11): 180419.
[28] Wanner S, Meister S, Goldluecke B. Datasets and benchmarks for densely sampled 4D light fields[M]//Bronstein M, Favre J, Hormann K. Vision, Modeling & Visualization, Lugano, Switzerland: The Eurographics Association, 2013: 225?226.
[29] Honauer K, Johannsen O, Kondermann D, et al. A dataset and evaluation methodology for depth estimation on 4D light fields[C]//Proceedings of the Asian Conference on Computer Vision, Taipei, Taiwan, China, 2016: 19?34.
[30] Raj S A, Lowney M, Shah R, et al. Stanford lytro light field archive[EB/OL]. 2016. http://lightfields.stanford.edu/LF2016.html.
[31] Rerabek M, Ebrahimi T. New light field image dataset[C]//Proceedings of the 8th International Conference on Quality of Multimedia Experience, Lisbon, Portugal, 2016.
[32] Chu X X, Zhang B, Ma H L, et al. Fast, accurate and lightweight super-resolution with neural architecture search[Z]. arXiv: 1901.07261, 2019.
[33] Kingma D P, Ba L J. Adam: a method for stochastic optimization[C]//Proceedings of the International Conference on Learning Representations, San Diego, America, 2015.
[34] Glorot X, Bengio Y. Understanding the difficulty of training deep feedforward neural networks[C]//Proceedings of the 13th International Conference on Artificial Intelligence and Statistics, Sardinia, Italy, 2010: 249?256.
[35] Bevilacqua M, Roumy A, Guillemot C, et al. Low-complexity single-image super-resolution based on nonnegative neighbor embedding[C]//British Machine Vision Conference, Guildford, UK, 2012.
[36] Chen J, Hou J H, Ni Y, et al. Accurate light field depth estimation with superpixel regularization over partially occluded regions[J]. IEEE Transactions on Image Processing, 2018, 27(10): 4889?4900.
基金项目：
国家自然科学基金资助项目(11772197)
导出参考文献，格式为：
引用本文：
赵圆圆, 施圣贤. 融合多尺度特征的光场图像超分辨率方法[J]. 光电工程, 2020, 47(12): 200007.
