No-reference point cloud quality assessment based on fusion of 3D and 2D features

Liu Taiwei; Yu Mei; Tu Renwei

doi:10.12086/oee.2025.250001

Article navigation > Opto-Electronic Engineering > 2025 Vol. 52 > No. 4 > 250001

Next Article Previous Article

Liu T W, Yu M, Tu R W. No-reference point cloud quality assessment based on fusion of 3D and 2D features[J]. Opto-Electron Eng, 2025, 52(4): 250001. doi: 10.12086/oee.2025.250001

Citation:

Liu T W, Yu M, Tu R W. No-reference point cloud quality assessment based on fusion of 3D and 2D features[J]. Opto-Electron Eng, 2025, 52(4): 250001. doi: 10.12086/oee.2025.250001

No-reference point cloud quality assessment based on fusion of 3D and 2D features

Faculty of Information Science and Engineering, Ningbo University, Ningbo, Zhejiang 315211, China

Fund Project: National Natural Science Foundation of China(62071266)

More Information

^*Corresponding author: yumei@nbu.edu.cn
CSTR: 32245.14.oee.2025.250001

Received Date 01 January 2025

Revised Date 26 February 2025

Accepted Date 28 February 2025

Published Date 25 April 2025

Abstract

Abstract

With the wide application of point clouds in virtual reality, computer vision, robotics and other fields, the assessment of distortions resulted from point cloud acquisition and processing is becoming an important research topic. Considering that the three-dimensional information of point clouds is sensitive to geometric distortion and the two-dimensional projection of point clouds contains rich texture and semantic information, a no-reference point cloud quality assessment method based on the fusion of three-dimensional and two-dimensional features is proposed to effectively combine the three-dimensional and two-dimensional feature information of point cloud and improve the accuracy of point cloud quality assessment. For 3D feature extraction, the farthest point sampling is firstly implemented on the point cloud, and then the non-overlapping point cloud sub-models centered on the selected points are generated, to cover the whole point cloud model as much as possible and use a multi-scale 3D feature extraction network to extract the features of voxels and points. For 2D feature extraction, the point cloud is first projected with orthogonal hexahedron projection, and then the texture and semantic information are extracted by a multi-scale 2D feature extraction network. Finally, considering the process of segmentation and interweaving fusion that occurs when the human visual system processes different types of information, a symmetric cross-modal attention module is designed to integrate 3D and 2D features. The experimental results on five public point cloud quality assessment datasets show that the Pearson’s linear correlation coefficient (PLCC) of the proposed method reaches 0.9203, 0.9463, 0.9125, 0.9164 and 0.9209 respectively, indicating that the proposed method has advanced performance compared with the existing representative point cloud quality assessment methods.
- point cloud quality assessment /
- 3D feature /
- 2D feature /
- symmetric cross-modal attention module

FullText(HTML)

References

[1]	陶志勇, 李衡, 豆淼森, 等. 融合多分辨率特征的点云分类与分割网络[J]. 光电工程, 2023, 50(10): 230166. doi: 10.12086/oee.2023.230166 CrossRef Google Scholar Tao Z Y, Li H, Dou M S, et al. Multi-resolution feature fusion for point cloud classification and segmentation network[J]. Opto-Electron Eng, 2023, 50(10): 230166. doi: 10.12086/oee.2023.230166 CrossRef Google Scholar
[2]	Gao W, Sun S, Zheng H, et al. OpenDMC: an open-source library and performance evaluation for deep-learning-based multi-frame compression[C]. Proceedings of the 31st ACM International Conference on Multimedia, 2023: 9685-9688 Google Scholar
[3]	Mekuria R N, Li Z, Tulvan C, et al. Evaluation criteria for PCC (point cloud compression)[J]. ISO/IEC JTC, 2016, 1: N16332. Google Scholar
[4]	Tian D, Ochimizu H, Feng C, et al. Geometric distortion metrics for point cloud compression[C]//2017 IEEE International Conference on Image Processing (ICIP), 2017: 3460–3464. https://doi.org/10.1109/ICIP.2017.8296925. Google Scholar
[5]	Cignoni P, Rocchini C, Scopigno R. Metro: measuring error on simplified surfaces[J]. Comput Graph Forum, 1998, 17(2): 167−174. doi: 10.1111/1467-8659.00236 CrossRef Google Scholar
[6]	Alexiou E, Ebrahimi T. Point cloud quality assessment metric based on angular similarity[C]//2018 IEEE International Conference on Multimedia and Expo (ICME), 2018: 1–6. https://doi.org/10.1109/ICME.2018.8486512. Google Scholar
[7]	Javaheri A, Brites C, Pereira F, et al. Improving psnr-based quality metrics performance for point cloud geometry[C]//2020 IEEE International Conference on Image Processing, 2020: 3438–3442. https://doi.org/10.1109/ICIP40778.2020.9191233. Google Scholar
[8]	Javaheri A, Brites C, Pereira F, et al. A generalized Hausdorff distance based quality metric for point cloud geometry[C]//2020 Twelfth International Conference on Quality of Multimedia Experience, 2020: 1–6. https://doi.org/10.1109/QoMEX48832.2020.9123087. Google Scholar
[9]	Javaheri A, Brites C, Pereira F, et al. Mahalanobis based point to distribution metric for point cloud geometry quality evaluation[J]. IEEE Signal Process Lett, 2020, 27: 1350−1354. doi: 10.1109/LSP.2020.3010128 CrossRef Google Scholar
[10]	Meynet G, Digne J, Lavoué G. PC-MSDM: a quality metric for 3D point clouds[C]//2019 Eleventh International Conference on Quality of Multimedia Experience (QoMEX), 2019: 1–3. https://doi.org/10.1109/QoMEX.2019.8743313. Google Scholar
[11]	Meynet G, Nehmé Y, Digne J, et al. PCQM: a full-reference quality metric for colored 3D point clouds[C]//2020 Twelfth International Conference on Quality of Multimedia Experience (QoMEX), 2020: 1–6. https://doi.org/10.1109/QoMEX48832.2020.9123147. Google Scholar
[12]	Yang Q, Ma Z, Xu Y L, et al. Inferring point cloud quality via graph similarity[J]. IEEE Trans Pattern Anal Mach Intell, 2020, 44(6): 3015−3029. doi: 10.1109/TPAMI.2020.3047083 CrossRef Google Scholar
[13]	Alexiou E, Ebrahimi T. Towards a point cloud structural similarity metric[C]//2020 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), 2020: 1–6. https://doi.org/10.1109/ICMEW46912.2020.9106005. Google Scholar
[14]	Hua L, Jiang G Y, Yu M, et al. BQE-CVP: blind quality evaluator for colored point cloud based on visual perception[C]//2021 IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB), 2021: 1–6. https://doi.org/10.1109/BMSB53066.2021.9547070. Google Scholar
[15]	Liu Q, Su H L, Duanmu Z F, et al. Perceptual quality assessment of colored 3D point clouds[J]. IEEE Trans Vis Comput Graph, 2023, 29(8): 3642−3655. doi: 10.1109/TVCG.2022.3167151 CrossRef Google Scholar
[16]	Lv J Y, Su H L, Long J C, et al. Point cloud quality assessment using multi-level features[J]. IEEE Access, 2024, 12: 47755−47767. doi: 10.1109/ACCESS.2024.3383536 CrossRef Google Scholar
[17]	Tu R W, Jiang G Y, Yu M, et al. Pseudo-reference point cloud quality measurement based on joint 2-D and 3-D distortion description[J]. IEEE Trans Instrum Meas, 2023, 72: 5019314. doi: 10.1109/TIM.2023.3290291 CrossRef Google Scholar
[18]	Liu Q, Yuan H, Su H L, et al. PQA-Net: deep no reference point cloud quality assessment via multi-view projection[J]. IEEE Trans Circuits Syst Video Technol, 2021, 31(12): 4645−4660. doi: 10.1109/TCSVT.2021.3100282 CrossRef Google Scholar
[19]	Yang Q, Liu Y P, Chen S H, et al. No-reference point cloud quality assessment via domain adaptation[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022: 21147–21156. https://doi.org/10.1109/CVPR52688.2022.02050. Google Scholar
[20]	Tao W X, Jiang G Y, Jiang Z D, et al. Point cloud projection and multi-scale feature fusion network based blind quality assessment for colored point clouds[C]//Proceedings of the 29th ACM International Conference on Multimedia, 2021: 5266–5272. https://doi.org/10.1145/3474085.3475645. Google Scholar
[21]	Tu R W, Jiang G Y, Yu M, et al. V-PCC projection based blind point cloud quality assessment for compression distortion[J]. IEEE Trans Emerg Top Comput Intell, 2023, 7(2): 462−473. doi: 10.1109/TETCI.2022.3201619 CrossRef Google Scholar
[22]	Tliba M, Chetouani A, Valenzise G, et al. PCQA-graphpoint: efficient deep-based graph metric for point cloud quality assessment[C]//ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023: 1–5. https://doi.org/10.1109/ICASSP49357.2023.10096610. Google Scholar
[23]	Qi C R, Yi L, Su H, et al. PointNet++: deep hierarchical feature learning on point sets in a metric space[C]. Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017: 5105–5114. Google Scholar
[24]	Dodge S, Karam L. Understanding how image quality affects deep neural networks[C]//2016 Eighth International Conference on Quality of Multimedia Experience (QoMEX), 2016: 1–6. https://doi.org/10.1109/QoMEX.2016.7498955. Google Scholar
[25]	He Z Y, Jiang G Y, Yu M, et al. TGP-PCQA: texture and geometry projection based quality assessment for colored point clouds[J]. J Vis Commun Image Represent, 2022, 83: 103449. doi: 10.1016/j.jvcir.2022.103449 CrossRef Google Scholar
[26]	Yang Q, Chen H, Ma Z, et al. Predicting the perceptual quality of point cloud: a 3D-to-2D projection-based exploration[J]. IEEE Trans Multimedia, 2021, 23: 3877−3891. doi: 10.1109/TMM.2020.3033117 CrossRef Google Scholar
[27]	Javaheri A, Brites C, Pereira F, et al. Point cloud rendering after coding: impacts on subjective and objective quality[J]. IEEE Trans Multimedia, 2021, 23: 4049–4064. https://doi.org/10.1109/TMM.2020.3037481. Google Scholar
[28]	Perry S, Cong H P, da Silva Cruz L A, et al. Quality evaluation of static point clouds encoded using MPEG codecs[C]//2020 IEEE International Conference on Image Processing (ICIP), 2020: 3428–3432. https://doi.org/10.1109/ICIP40778.2020.9191308. Google Scholar
[29]	Alexiou E, Viola I, Borges T M, et al. A comprehensive study of the rate-distortion performance in MPEG point cloud compression[J]. APSIPA Trans Signal Inf Process, 2019, 8(1): e27. doi: 10.1017/ATSIP.2019.20 CrossRef Google Scholar

Overview

Overview

Point clouds are widely used in virtual reality, computer vision, robotics and other fields, and distortion assessment in point cloud acquisition and processing is becoming an important research topic. Considering that the three-dimensional (3D) information of point cloud is sensitive to geometric distortion and the two-dimensional (2D) projection of point cloud contains rich texture and semantic information, this paper proposes a no-reference point cloud quality assessment method to effectively combine the 3D and 2D feature information of point cloud and improve the accuracy of quality assessment. The farthest point sampling is firstly implemented on the point cloud, and then the non-overlapping point cloud sub-models centered on the selected points are generated, to cover the whole point cloud model as much as possible. For each point cloud sub-model, an improved 3D multi-scale feature extraction network (MSFNet) is designed to extract the features of voxels and points. MSFNet contains three point-voxel transformer (PVT) modules and generates output features through a multilayer perceptron. Each PVT module has two branches. The voxel branch can extract rich semantic features from spatial voxels; the point-based branch can retain the integrity of the point cloud sub-model position information as much as possible and avoid the loss of position information. For 2D feature extraction, the point cloud is first projected with orthogonal hexahedron projection to obtain the corresponding projection maps. To extract the rich texture and semantic information from the 2D projection maps, a 2D multi-scale feature extraction network (MSTNet) is designed to extract 2D content-aware features. Considering that there may be a large amount of redundant information and certain dependency relationships between different viewpoint projection maps, MSTNet uses spatial global average pooling operation to remove redundant information and spatial global standard deviation pooling operation to preserve the dependency information between different viewpoint projection maps. Finally, considering the process of segmentation and interweaving fusion that occurs when the human visual system processes different modality information, to better fuse the 2D and 3D features of the point cloud, so that the two modality features can enhance each other, a symmetric cross-modality attention module is designed to integrate the 3D and 2D features, and a multi-head attention mechanism is added in the feature fusion process. The experimental results on five public point cloud quality assessment datasets show that the Pearson's linear correlation coefficient (PLCC) of the proposed method reaches 0.9203, 0.9463, 0.9125, 0.9164, and 0.9209, respectively, indicating that the proposed method has advanced performance compared with the existing representative point cloud quality assessment methods.