Multi-resolution feature fusion for point cloud classification and segmentation network

Tao Zhiyong; Li Heng; Dou Miaosen; Lin Sen

doi:10.12086/oee.2023.230166

Abstract

To address the problem that existing networks find it difficult to learn local geometric information of point cloud effectively, a graph convolutional network that fuses multi-resolution features of point cloud is proposed. First, the local graph structure of the point cloud is constructed by the k-nearest neighbor algorithm to better represent the local geometric structure of the point cloud. Second, a parallel channel branch is proposed based on the farthest point sampling algorithm, which obtains point clouds with different resolutions by downsampling them and then groups them. To overcome the sparse characteristics of the point cloud, a geometric mapping module is proposed to perform normalization operations on the grouped point cloud. Finally, a feature fusion module is proposed to aggregate graph features and multi-resolution features to obtain global features more effectively. Experiments are evaluated using ModelNet40, ScanObjectNN, and ShapeNet Part datasets. The experimental results show that the proposed network has state-of-the-art classification and segmentation performance.

Keywords

FullText(HTML)

1. 引　言

随着激光雷达等3D扫描技术的快速发展^[1-2]，点云的获取变得容易，近年来，点云分析已经成为3D视觉任务的热门话题，在自动驾驶^[3-4]、医学影像^[5]、机器人^[6]、三维重建^[7]等领域得到了广泛应用。尽管深度学习在点云处理方面取得了不错的成果，但是由于点云无序且不规则的特性，有效处理点云的数据仍然具有挑战性。

将神经网络适用于点云数据并非易事。早期的点云处理主要分为基于体素的方法和基于多视图的方法。前者将点云转换为体素，通过规则的3D网格结构表征点云。如：Maturana等人^[8]提出了VoxNet，通过体素化点云以实现3D形状的识别任务；Riegler等人^[9]提出了OctNet，对八叉树结构进行有效编码，并对每个体素的特征向量进行简单的算法索引；文献[10]针对传统体素网格降采样存在采样点不均匀的问题，提出了一种新的体素网格降采样方式，提高了计算效率。但随着分辨率的增加，这类方法的计算量和内存占用量会以立方形式增长。基于多视图的方法是将点云从不同角度投影至多个平面，使其适用于2D卷积。如：Su等人^[11]提出了MVCNN，将多视图特征最大池化为一个全局描述符，但是最大池化操作只保留了特定视图中的最大元素，不可避免地导致信息缺失。为了不引入中间表示(体素和多视图)来直接处理点云数据，基于点的方法得到了发展。Qi等人^[12]提出的PointNet是该类方法的先驱，通过几个多层感知机(MLP)独立地学习逐点特征，克服了点云的无序性，但同时却忽略了点对之间的局部关系。为此，Qi等人^[13]扩展了PointNet，提出新的网络PointNet++，通过在不同的层次上使用PointNet提取特征，但是本质上还是在独立地处理局部区域的每一个点，点对之间的几何联系仍然被忽略。点云作为一种不规则的非欧式数据，类图结构能对其进行有效地表征。Simonovsky等人^[14]是点云图论的先驱，将每一个点视为图的顶点，通过有向边将顶点和其邻域点进行连接，然后基于滤波器和MLP提出了边缘条件卷积(edge-conditioned convolution, ECC)。Wang等人^[15]在DGCNN中提出一种边缘卷积算子(edge convolution, EdgeConv)，通过在特征空间中构造局部图结构表示点云的局部联系，并在网络的每一层之后动态更新图特征。Zhang等人^[16]基于DGCNN提出了LDGCNN，删去DGCNN中的变换网络并将不同层次的特征进行连接，以提高网络的性能。文献[17]通过引入位置关系改进DGCNN，强化局部特征的提取能力。此外，一些方法通过引入注意力机制来提升网络性能。如：Sun等人^[18]提出了基于双向注意力机制的残差图卷积网络，以更好地区分任务相关特征；Gou等人^[19]基于transformer提出了名为PCT的网络执行点云分析任务；Liu等人^[20]将SENet和注意力机制引入PointNet++，优化了网络对于重要特征的学习能力。

为了能更有效地提取点云局部特征，提高网络分类与分割性能，提出一种融合多分辨率特征的图卷积网络。首先考虑到点云的无序不规则特性，通过k-最近邻算法(k-nearest neighbor, kNN)在特征空间中构造局部图结构，通过图结构表征点云局部区域中点对之间的几何联系。其次，通过最远点采样算法(farthest point sampling, FPS)对点云进行多级下采样操作，以获得不同分辨率的点云，考虑其局部区域的几何关系，采用kNN算法进行分组；为克服点云的稀疏性质，引入几何映射模块将分组点集正态化。最后，为了实现良好的分类与分割性能，提出一种特征融合方式，用点云多分辨率特征对局部图特征进行补偿。通过在ModelNet40^[21]、ScanObjectNN^[22]和ShapeNet Part^[23]数据集上对网络的分类、分割性能进行评估，验证本文方法的有效性。所提出的多分辨率图卷积模块 (multi-resolution graph convolution module, M-R GCM)算法流程如图1所示。

Figure 1. Full-Size Img PowerPoint

Multi-resolution graph convolution module algorithm flow chart

2. 原理与方法

2.1 网络整体框架

网络框架如图2所示，输入是点数为N，维度为3的点云数据。受启发于DGCNN模型^[15]，并以其为基线网络，去除变换网络，通过融合多分辨率特征对EdgeConv进行改进。分类网络用四个多分辨率图卷积模块学习点云上下文局部信息，为克服点云的无序特性，从而引入池化策略(pooling strategy)，其中最大池化(max pooling)和平均池化(avg pooling)分别保留特征图中的最显著特征以及整体特征，然后对其进行拼接后获得全局特征作为分类器的输入。最后由全连接层(fully connected layer)作为分类器输出分类得分C；分割网络主干包含三个多分辨率图卷积模块，池化策略选择max pooling，最终输出P个语义标签作为每个点的得分。

Figure 2. Full-Size Img PowerPoint

Network framework. (a) Classification network; (b) Segmentation network

2.2 图卷积分支

与基于图方法的DGCNN^[15]类似，将类图结构应用于点云特征学习可以更为有效地处理点云这类非结构化数据，其核心在于选取点云表面的点作为节点，并与邻域点之间建立边，从而构建局部点对之间的几何联系。图卷积即为对该图结构进行卷积操作，具体过程如图3所示。

Figure 3. Full-Size Img PowerPoint

The operation procedure of graph convolution

首先，定义输入的点云为包含n个点的点集 ${\boldsymbol{ P}}$ $= {\left\{ {{{\boldsymbol{p}}_1},{{\boldsymbol{p}}_2},...,{{\boldsymbol{p}}_i}} \right\}_{i = 1,2,...,n}}$ $\subseteq {{\bf{R}}^F}$ ，其中R为实数集合，F为点云的维度，一般取F=3，即每一个点 ${{\boldsymbol{p}}_i}$ 用其空间坐标 $({x_i},{y_i},{{\textit{z}}_i})$ 表示，此外还可以包括RGB信息、法线等额外的维度信息。由于在神经网络中，下一层的输入一般为上一层的输出，因此一般而言F表示给定的特征维度。

然后，在点云的表面构建有向图G，表示为

${\boldsymbol{G }}= ({\boldsymbol{V}},{\boldsymbol{E}}) \;,$

其中：V= $\left\{ {{{\boldsymbol{p}}_1},{{\boldsymbol{p}}_2},...,{{\boldsymbol{p}}_n}} \right\}$ 和 ${\boldsymbol{E}} \subseteq {\boldsymbol{V}} \times {\boldsymbol{V}}$ 分别为有向图的顶点和边。在图卷积模块中，采用kNN算法对每一个顶点 ${{\boldsymbol{p}}_i}$ 进行检索，找到其周围的k个邻域点构成邻域点集，表示为

${{\boldsymbol{N}}_i} = {\left\{ {{{\boldsymbol{p}}_{i,1}},{{\boldsymbol{p}}_{i,2}},...,{{\boldsymbol{p}}_{i,j}}} \right\}_{j = 1,2,...,k}} \;,$

其中， ${{\boldsymbol{p}}_{i,j}}$ 为节点 ${{\boldsymbol{p}}_i}$ 的第j个邻域点，通过计算 ${{\boldsymbol{p}}_i}$ 与 ${\{ {{\boldsymbol{p}}_{i,j}}\} _{j = 1,2,...,k}}$ 之间的欧几里得距离作为图G的边，表示为

${{\boldsymbol{e}}_{i,j}} = {\{ {{\boldsymbol{p}}_i} - {{\boldsymbol{p}}_{i,j}}\} _{j = 1,2,...,k}} \;.$

由式(3)计算边特征为

${{\boldsymbol{e}}_i} = {h_\varTheta }\{ {{\boldsymbol{p}}_i},{{\boldsymbol{p}}_i} - {{\boldsymbol{p}}_{i,j}}\} \;,$

其中， ${h_\varTheta }$ 是一个包含一组可学习参数 $\varTheta$ 的非线性函数，可实现特征维度R^F×R^F→R^F。此外，由于对h的选择方式不同，边特征的计算方式也不同，其余计算方式如

${{\boldsymbol{e}}_i} = {h_\varTheta }\{ {{\boldsymbol{p}}_i}\} \;,$

${{\boldsymbol{e}}_i} = {h_\varTheta }\{ {{\boldsymbol{p}}_{i,j}}\} \;.$

式(5)为PointNet^[12]的方式，只考虑每一个点，忽视点云的局部结构，式(6)为只考虑了点云的局部关系却忽视全局结构。因此本文采用式(4)所示的方法，由 ${{\boldsymbol{p}}_i}$ 和 $\{ {{\boldsymbol{p}}_i} - {{\boldsymbol{p}}_{i,j}}\}$ 同时兼顾全局和局部结构。

最后，通过聚合函数对特征 ${\{ {{\boldsymbol{e}}_i}\} _{i = 1,2,...,n}}$ 进行操作，将更新后的图特征 ${{\boldsymbol{f}}_{\rm{G}}}$ 表示为

${{\boldsymbol{f}}_{\rm{G}}} = \mathop \square \limits_{j:(i,j) \in {\boldsymbol{E}}} \{ {h_\varTheta }({{\boldsymbol{p}}_i},{{\boldsymbol{e}}_{i,j}})\} \;,$

其中： $\square$ 表示Max pooling策略， ${h_\varTheta }$ 为MLP操作，表示为 ${h_\varTheta }({{\boldsymbol{p}}_i},{\boldsymbol{e}_{i,j}}) = LeakyReLU{\text{(}}{\theta _i} \cdot {{\boldsymbol{p}}_i} + {\phi _i} \cdot {{\boldsymbol{e}}_{i,j}}{\text{)}}$ ，可学习参数 $\varTheta = ({\theta _1},...,{\theta _n},{\phi _1},...,{\phi _n})$ 。

考虑到图卷积网络的性能受限于预定义邻域的大小(kNN中k的取值，实验部分将进行验证)，本文考虑用不同分辨率的点云特征对其进行融合补偿，从而提高网络的性能。

2.3 多分辨率分支

在图像处理任务中，神经网络层数的增加能更为有效地提取复杂特征，但往往会以牺牲空间分辨率的方式来换取特征通道数的增加。本文从弥补空间分辨率的角度，设计多分辨率分支对基线网络DGCNN的图卷积策略进行改进，从而提高模型性能。多分辨率点云特征的学习过程如图4所示。

Figure 4. Full-Size Img PowerPoint

The process of learning multi-resolution point cloud features

该分支对点云的处理分为三个阶段，第一个阶段为基于FPS算法的下采样阶段(down sampling)。当输入为包含n个点的点云 ${\boldsymbol{X}} \subseteq {{\bf{R}}^d}$ ，其中d为点云的特征维度，每一个点表示为 ${\{ {{\boldsymbol{x}}_i}\} _{i = 1,2,...,n}} \in \boldsymbol{X}$ ，选择其中的一个点 ${{\boldsymbol{x}}_1}$ 作为起始点得到第一个采样点集合，表示为 ${\boldsymbol{S}} = \left\{ {{{\boldsymbol{x}}_1}} \right\}$ ；然后计算所有点与x₁的距离，用数组 $L = ({l_1},{l_2},{l_i},...)$ 进行储存，选择其中距离最远的点 ${{\boldsymbol{x}}_1}$ ，更新采样点集合为 $\boldsymbol{S} = \left\{ {{{\boldsymbol{x}}_1},{{\boldsymbol{x}}_2}} \right\}$ ；再计算所有点与 ${{\boldsymbol{x}}_1}$ 的距离，若其中的一个点x_i与其的距离小于 ${l_i}$ ，则将 ${l_i}$ 进行更新，数组L始终记录最小距离；选择其中距离最远的点 ${{\boldsymbol{x}}_3}$ 更新采样点集合为 ${\boldsymbol{S}} = \left\{ {{{\boldsymbol{x}}_1},{{\boldsymbol{x}}_2},{{\boldsymbol{x}}_3}} \right\}$ ，重复采样 $n'$ 次后，得到的采样点集表示为

${\boldsymbol{S}} = \left\{ {{{\boldsymbol{x}}_1},{{\boldsymbol{x}}_2},{{\boldsymbol{x}}_3},...,{{\boldsymbol{x}}_{n'}}} \right\} \;.$

上述操作在改变点云分辨率的同时，新的采样点集S仍然能够较好地表征点云的表面形状。第二个阶段为局部分组阶段(local grouping)，在对不同分辨率点云进行处理时，考虑其局部几何关系，采用kNN算法对式(8)的点集进行局部组划分，每一个局部组点集表示为

${{\boldsymbol{S}}_i} = \left\{ {{{\boldsymbol{x}}_{i,1}},{{\boldsymbol{x}}_{i,2}},...,{{\boldsymbol{x}}_{i,k}}} \right\} \;.$

考虑点云的稀疏特性，为提高网络的性能，在第三个阶段(normalization)中，对每一个局部组 ${\boldsymbol{S}_i}$ 正态化，具体操作如下

${{\boldsymbol{S}}_i}' = \frac{{{{\boldsymbol{S}}_i} - {\text{mean}}({{\boldsymbol{S}}_i})}}{{\sqrt {bias} }} \;,$

$bias = \frac{{{{\displaystyle \sum\limits_{i = 1}^{n'} {\displaystyle \sum\limits_{j = 1}^k {\left( {{\boldsymbol{x}}_{i,j}^{} - {\boldsymbol{x}}_i^{}} \right)^2} } }}}}{{k \times n' \times d}} \;,$

其中：mean为求均值操作，bias表示局部组中元素的方差，k为局部组中邻域点数量。因此，多分辨率特征表示为

${{\boldsymbol{f}}_{{\rm{MR}}}} = LeakyReLU\left( {{\boldsymbol{a}} \odot {{\boldsymbol{S}}_i}' + {\boldsymbol{b}}} \right) \;,$

式中：a∈R^d和b∈R^d是两个可学习参数，⊙表示Hadamard乘积。

并且，多分辨率分支作为并行分支而不是串行分支连接进网络。在每个多分辨率图卷积模块输出特征之前，特征融合模块基于图卷积分支和多分辨率分支的特征，通过网络计算获得一个自适应权重系数对两路特征进行加权输出，并作为下一个多分辨率图卷积模块的输入，在之后的每一层网络中动态更新，直至进入全连接层进行分类。在整个过程两个分支不断进行特征交互，以此实现特征图更新。

2.4 特征融合模块

为了使多分辨率特征更有效地对图特征进行补偿，提出了一种特征融合模块，相比较于线性特征聚合方式(如加法运算)，该融合模块更具特征的自适应性。特征融合原理如图5所示。

Figure 5. Full-Size Img PowerPoint

The operation of feature fusion

在每一个多分辨率图卷积模块中，通过两个分支的网络分别学习到图特征 ${{\boldsymbol{f}}_{\rm{G}}}$ 与多分辨率特征 ${{\boldsymbol{f}}_{\rm{MR}}}$ ，为更有效地集成特征，首先将两路特征进行逐元素求和，以紧凑地表示点云特征f：

${\boldsymbol{f}} = {{\boldsymbol{f}}_{\rm{G}}} + {{\boldsymbol{f}}_{\rm{MR}}} \;.$

接着基于该特征矩阵生成一个权重向量，表示为

${\boldsymbol{\alpha}} = \sigma \left( {\Phi ({\boldsymbol{f}})} \right) \;,$

式中： $\sigma$ 为Sigmoid激活函数， $\Phi$ 为MLP操作。最终由该权重向量对两个分支的特征进行加权输出，特征图表示为

${\boldsymbol{f}} ' = {\boldsymbol{\alpha}} \odot {{\boldsymbol{f}}_{\rm{G}}} + ({\boldsymbol{1 }}- \boldsymbol{\alpha} ) \odot {{\boldsymbol{f}}_{\rm{MR}}} \;,$

其中：1为全元素为1的向量，⊙表示Hadamard乘积。

3. 实验结果与分析

3.1 实验环境配置

实验参数设置如表1所示。为验证本文方法的有效性，在ModelNet40^[21]、ScanObjectNN^[22]数据集上进行分类实验，在ShapeNet Part^[23]数据集上进行部分分割实验。实验参数设置如表1所示，所有实验均基于Linux Ubuntu系统，训练所用GPU为GeForce RTX 3090，学习框架为Python3.7+Pytorch-1.7.1。

Experimental parameter setting

实验参数设置

参数项	分类网络	分割网络
输入点数	1024	2048
多分辨率点云点数	[896,768,640,512]	[896,768,640]
图卷积分支k取值	20	20
训练周期	250	300
优化器	SGD	SGD
训练批次	32	32
测试批次	16	16
初始学习速率	0.1	0.003

CSV Show Table

3.2 基于ModelNet40数据集的点云分类

3.2.1 数据集描述

ModelNet40数据集包含40个类别的12311个CAD模型，其中9843个用于训练，2468个用于测试。在训练和测试期间没有执行任何数据预处理操作，保证实验的有效性。

3.2.2 分类结果与分析

在ModelNet40数据集上的分类结果如表2所示，评价指标为总体准确率(overall accuracy, OA)和类平均准确率(mean class accuracy, mAcc)。

Comparison of classification accuracy with different methods on ModelNet40 dataset

不同方法在ModelNet40数据集上的分类精度对比

方法	输入	点数/10³	mAcc/%	OA/%
VoxNet^[8]	体素	-	83.0	85.9
MVCNN^[11]	多视图	-	-	90.1
PointNet^[12]	坐标	1	86.0	89.2
PointNet++^[13]	坐标+法线	5	-	91.9
文献[24]	坐标+法线	1	89.8	91.6
文献[25]	坐标+法线	1	-	93.0
3D-GCN^[26]	坐标	1	-	92.1
DGCNN^[15]	坐标	1	90.2	92.9
LDGCNN^[16]	坐标	1	90.3	92.9
DDGCN^[27]	坐标	1	90.4	92.7
DRNet^[28]	坐标	1	-	93.1
DGANet^[29]	坐标	1	89.4	92.3
PCT^[19]	坐标	1	-	93.2
AFM-Net^[30]	坐标	1	89.4	92.85
文献[31]	坐标	1	89.02	92.5
Our	坐标	1	91.2	93.4

CSV Show Table

$OA = \frac{{TP}}{{TP + FN}} \;,$

$mAcc = \frac{1}{c}\sum\limits_{i = 1}^c {\frac{{T{{{P}}_i}}}{{T{{{P}}_i} + F{N_i}}}} \;,$

其中：TP表示正确预测正样本的数目，FN为错误预测正样本的数目，而TP_i和FN_i分别表示每一个类别中正样本被预测正确和错误的样本数。

从表2中可以看出，对比基于体素的经典方法VoxNet和基于多视图的MVCNN，本文的方法在OA这个指标上提升了7.5%和2.7%；与PointNet和PointNet++相比也有4.2%和1.5%的提升，并和基于图的网络DGCNN、LDGCNN、DGANet相比分别提升了0.5%、0.5%和1.1%。与新颖的方法如PCT、AFM-Net以及文献[31]相比，也有一定的精度提升。实验结果表明，改进后的多分辨率图卷积网络在分类任务中具有一定的优越性。

3.3 基于ScanObjectNN数据集的点云分类

3.3.1 数据集描述

ScanObjectNN数据集是一个以真实世界对象为模型的数据集，包含15个类别的15000个对象，其中有2902个唯一对象实例。由于模型中存在复杂背景、噪声以及遮挡，该数据集对于现有点云分类任务更具有挑战性。

3.3.2 分类结果与分析

基于ScanObjectNN数据集的分类结果如表3所示，所有模型的输入均为坐标形式，本文模型的OA和mAcc分别达到了83.3%、81.7%。从表3中可以看出，本文的方法在评估指标上比PointNet提高了15.1%(OA)、18.3%(mAcc)；与基线网络DGCNN相比提高了5.2%(OA)、8.1%(mAcc);与新颖的方法GBNet相比提高2.8%(OA)、3.9%(mAcc)。实验结果表明，当数据集处于噪声干扰的情况下，本文提出的方法通过弥补一定的空间分辨率，能够更好地识别形状特征，比起其它方法具有更强的形状分类能力。

Comparison of classification accuracy with different methods on ScanObjectNN dataset

不同方法在ScanObjectNN数据集上的分类精度对比

方法	输入	mAcc/%	OA/%
PointNet^[12]	坐标	63.4	68.2
PointNet++^[13]	坐标	75.4	77.9
DGCNN^[15]	坐标	73.6	78.1
DRNet^[28]	坐标	78.0	80.3
GBNet^[32]	坐标	77.8	80.5
PRANet^[33]	坐标	79.1	82.1
Ours	坐标	81.7	83.3

CSV Show Table

3.4 基于ShapeNet Part数据集的点云分割

3D点云的部分分割是一项具有挑战性的细粒度识别任务。简单而言，分割任务需要为每个点分配类别标签，如机翼、机身等。为验证模型的分割性能，在ShapeNet Part数据集上进行部分分割实验。

3.4.1 数据集描述

ShapeNet Part数据集由16个类别的16881个模型组成，共标记了50个零件标签，其中13998个模型用于训练，2874个用于测试，每一个模型分为2到6个不同的部分进行标注。训练时从每个形状中采样2048个点。

3.4.2 分割结果与分析

以ShapeNet Part为基准数据集，部分分割的结果如表4所示，分割实验的评价指标为平均交并比(mean intersection over union, mIoU)。

Part segmentation results on the ShapeNet Part dataset

ShapeNet Part数据集上的部分分割结果

方法	PointNet^[12]	PointNet++^[13]	文献[25]	3D-GCN^[26]	LDGCNN^[16]	DGANet^[29]	DGCSA^[34]	DGCNN^[15]	本文
飞机	83.4	82.4	83.8	83.1	84.0	84.6	84.2	84.0	83.6
包	78.7	79.0	77.5	84.0	83.0	85.7	73.3	83.4	83.4
帐篷	82.5	87.7	87.9	86.6	84.9	87.8	82.3	86.7	88.4↑
车	74.9	77.3	78.7	77.5	78.4	78.5	77.7	77.8	78.4↑
椅子	89.6	90.8	90.8	90.3	90.6	91.0	91.0	90.6	89.7
耳机	73.0	71.8	77.3	74.1	74.4	77.3	75.3	74.7	80.5↑
吉他	91.5	91.0	91.8	90.9	91.0	91.2	91.2	91.2	91.8↑
刀	85.9	85.9	87.9	86.4	88.1	87.9	88.6	87.5	88.6↑
台灯	80.8	83.7	84.2	83.8	83.4	82.4	85.3	82.8	81.6
手提电脑	95.3	95.3	95.9	95.6	95.8	95.8	95.9	95.7	95.8↑
摩托	65.2	71.6	71.8	66.8	67.4	67.8	58.9	66.3	69.6↑
马克杯	93.0	94.1	95.1	94.8	94.9	94.2	94.3	94.9	94.4
手枪	81.2	81.3	80.9	81.3	82.3	81.1	81.8	81.1	83.7↑
火箭	57.9	58.7	59.6	59.6	59.2	59.7	56.9	63.5	62.5
滑板	72.8	76.4	76.6	75.7	76.0	75.7	75.4	74.5	82.0↑
桌子	80.6	82.6	82.4	82.8	81.9	82.0	82.7	82.6	83.0↑
mIoU	83.7	85.1	85.4	85.1	85.1	85.2	85.3	85.2	85.4↑

CSV Show Table

$mIoU = \frac{{TP}}{{TP + FP + FN}} \;,$

其中：TP表示对正样本预测正确的数目，FP为对负样本预测错误的数目，FN为对正样本预测正确的数目。本文模型在部分分割任务中，达到了85.4%的精度。对比经典算法PointNet、PointNet++分别提高了1.6%和0.2%。并且与基于图结构的算法DGCNN、LDGCNN和DGANet相比，多分辨率图卷积网络的精度分别提高了0.1%、0.2%和0.1%。在该实验中，有10个类别的物体分割精度较于基线网络DGCNN均有所改善(“↑”代表有改善的物体类别)，实验结果表明，多分辨率图卷积网络在分割任务中也实现了一定的优化。

为了能更直观地展示本文模型的分割效果，将16个类别的物体进行分割结果可视化如图6所示。同时，将DGCNN作为基线网络，展示了经本文方法改进后部分分割的细节对比如图7所示，红色方框内效果差异。

Figure 6. Full-Size Img PowerPoint

The results of the part segmentation visualization. (a) Groud truth; (b) Ours

Figure 7. Full-Size Img PowerPoint

Comparison of segmentation details. (a) Groud truth; (b) Ours; (c) Baseline

通过上述定量、定性分析比较，融合多分辨率特征的图卷积网络能正确分割大部分点，并且对比基线网络，本文方法在目标边界处的分割能力也有所提升，相较于改进之前的网络有一定的优越性。

4. 消融实验

为验证网络的有效性和鲁棒性，以ModelNet40为基准数据集，对模型各模块及超参数的选择进行分析验证。

4.1 图卷积分支中k取值对性能的影响

回顾2.2小节，局部图结构基于kNN算法构造，该预定义邻域的大小影响模型的性能，通过改变k的取值进行实验以寻找合适的邻域大小。实验结果如表5所示。

Effect of different k values on model performance

不同k值对模型性能的影响

k	OA(%)	用多分辨率分支补偿后OA/%	提升/%
5	20.7	35.1	+14.4
10	85.4	88.3	+2.9
15	91.9	92.1	+0.2
20	92.5	93.4	+0.9
25	92.1	92.3	+0.2

CSV Show Table

实验表明，当k值过小或过大都会导致网络性能下降，取值为20的时候性能最佳。并且由表5可以看出，当引入多分辨率分支对图特征进行补偿后，网络的性能都得到了提升，证明了多分辨率分支的有效性。

4.2 多分辨率图卷积模块的消融实验

在本节中设置四组实验以验证多分辨率图卷积模块各部分对网络性能的影响，实验结果如表6所示。

Ablation experiments of multi-resolution GCN module

多分辨率图卷积模块消融实验

实验	GCN分支	M-R分支	融合	mAcc/%	OA/%
1	√	×	×	89.9	92.5
2	×	√	×	84.0	89.1
3	√	√	×	89.9	92.6
4	√	√	√	91.2	93.4

CSV Show Table

实验1：基线网络，仅使用图卷积分支进行特征学习；

实验2：仅使用多分辨率分支进行特征学习；

实验3：使用两个分支进行特征学习，但特征集成方式采用线性组合方式；

实验4：在实验3的基础上将特征集成方式替换为所提出的特征融合模块。

实验结果表明，同时使用两个分支的网络性能均优于使用单一分支，并且当采用所提出的特征融合模块来聚合两个分支的特征后，网络的精度进一步获得提升。

4.3 多分辨率点云规模的选取

在本文方法中，点云的空间分辨率影响网络的性能，每一个多分辨率图卷积模块中都存在图特征与不同分辨率特征的交互，表7展示了不同分辨率的点云对网络性能的影响。

The effect of different resolution point cloud on network performance

不同分辨率点云对网络性能的影响

不同分辨率的点云	mAcc/%	OA/%
[512,384,256,128]	90.4	92.7
[640,512,384,256]	90.6	92.8
[768,640,512,384]	90.9	93.0
[896,768,640,512]	91.2	93.4

CSV Show Table

实验结果表明，当四个多分辨率图卷积模块中不同分辨率点云的点数为[896,768,640,512]时，模型的性能达到最优。

4.4 噪声鲁棒性测试

现实生产应用中，通过扫描设备所获取的点云数据往往存在噪声干扰，为测试网络对于噪声的鲁棒性，对初始点云添加不同水平的随机噪声模拟实际生产应用的情况，实验结果如图8所示。

Figure 8. Full-Size Img PowerPoint

Noise robustness testing

图8展示了在不同水平噪声干扰下本文方法与基线网络 DGCNN 以及基于图方法的3D-GCN、AdaptConv^[35]的精度对比，可以看出在噪声干扰下，本文方法的性能总体上优于DGCNN，与其它两种模型对比也有一定优势。表8展示了两种方法在噪声干扰下精度下降的程度，下降程度的计算公式如式(19)：

Comparison of the noise robustness of the several methods

多种模型的噪声鲁棒性比较

噪声水平	下降程度
噪声水平	3D-GCN	AdaptConv	DGCNN	Ours
0.02	0.7↓	1.8↓	1.4↓	0.9↓
0.04	2.2↓	2.2↓	2.2↓	1.8↓
0.06	4.6↓	3.3↓	3.2↓	3.1↓
0.08	8.4↓	6.5↓	5.7↓	6.4↓
0.1	14.9↓	10.8↓	13.1↓	11.7↓

CSV Show Table

$\downarrow \%=\frac{OA(噪声下)-OA(无噪声)}{OA(无噪声)}.$

结合图表分析，多分辨率图卷积网络的噪声鲁棒性对比基线网络得到了提高，在抗噪能力上具有一定的优越性。

4.5 特征提取模块数量消融实验

在图2的分类网络框架中，由四个局部特征提取模块(多分辨率图卷积模块)组成，在本节实验中，将使用不同数量的模块以寻找最优的模型框架，实验结果如表9所示。

The impact of different number of feature extraction modules on network performance

不同数量特征提取模块对网络性能的影响

模块数量	mAcc/%	OA/%	每轮训练时间/s	模型参数量/M
3	89.7	92.4	63	2.8
4	91.2	93.4	139	3.6
5	90.6	93.1	323	4.8

CSV Show Table

在上述实验中，使用三个模块时的点云分辨率为[896,768,640]，四个模块为[896,768,640,512]，五个模块为[896,768,640,512,384]，分析实验结果可知特征提取模块的数量越多，网络的性能并不会一直上升，反而导致模块复杂度与所需的内存占用增加。因此，为兼顾精度与效率，选择四个特征提取模块最为合适。

5. 结　论

本文在图卷积网络DGCNN的基础上提出了一种融合多分辨率特征的图卷积模块，通过引入不同分辨率的点云特征对传统的图结构信息进行补偿，克服传统图结构信息受限于预定义邻域规模的问题，用空间分辨率丰富了点云的局部特征。并且相较于一般的特征线性聚合方式，所提出的特征聚合模块更具有自适应性，能更加有效地进行点云特征学习。通过在ModelNet40、ScanObjectNN以及ShapeNet Part三个具有挑战性的基准数据集上进行分类分割实验，证明融合多分辨率特征的图卷积网络性能相比于其它模型均有一定提升。此外，大量消融实验也验证了多分辨率图卷积网络在点云分析任务中的有效性和鲁棒性。

虽然，融合多分辨率特征的图卷积网络在基准数据集上取得了令人满意的结果，但在实际应用中，样本的分布并不总是均匀的，并且该网络在分割实验中，样本边界的识别能力也有所欠缺，这些都是本文网络将来的优化方向。今后的工作将聚焦于捕获点云细粒度特征的研究前沿，进一步提高网络在复杂场景下的识别能力，并与实际生产应用相结合。

利益冲突

所有作者声明无利益冲突

References (35)

References

[1]	张昕怡, 陈茂霖, 刘祥江, 等. 顾及点密度与未知角分辨率的地面点云分类[J]. 激光技术, 2023, 47(1): 59−66. DOI: 10.7510/jgjs.issn.1001-3806.2023.01.009 Zhang X Y, Chen M L, Liu X J, et al. Classification of terrestrial point cloud considering point density and unknown angular resolution[J]. Laser Technol, 2023, 47(1): 59−66. DOI: 10.7510/jgjs.issn.1001-3806.2023.01.009 CrossRef Google Scholar
[2]	李佳男, 王泽, 许廷发. 基于点云数据的三维目标检测技术研究进展[J]. 光学学报, 2023, 43(15): 1515001. DOI: 10.3788/AOS230745 Li J N, Wang Z, Xu T F. Three-dimensional object detection technology based on point cloud data[J]. Acta Opt Sin, 2023, 43(15): 1515001. DOI: 10.3788/AOS230745 CrossRef Google Scholar
[3]	陆康亮, 薛俊, 陶重犇. 融合空间掩膜预测与点云投影的多目标跟踪[J]. 光电工程, 2022, 49(9): 220024. DOI: 10.12086/oee.2022.220024 Lu K L, Xue J, Tao C B. Multi target tracking based on spatial mask prediction and point cloud projection[J]. Opto-Electron Eng, 2022, 49(9): 220024. DOI: 10.12086/oee.2022.220024 CrossRef Google Scholar
[4]	陈仲生, 李潮林, 左旺, 等. 双重下采样增强的点云改进配准算法研究[J]. 汽车工程, 2023, 45(4): 572−578. DOI: 10.19562/j.chinasae.qcgc.2023.04.005 Chen Z S, Li C L, Zuo W, et al. Study on improved point cloud registration algorithm enhanced by double down-sampling[J]. Automot Eng, 2023, 45(4): 572−578. DOI: 10.19562/j.chinasae.qcgc.2023.04.005 CrossRef Google Scholar
[5]	李美佳, 于泽宽, 刘晓, 等. 点云算法在医学领域的研究进展[J]. 中国图象图形学报, 2020, 25(10): 2013−2023. DOI: 10.11834/jig.200253 Li M J, Yu Z K, Liu X, et al. Progress of point cloud algorithm in medical field[J]. J Image Graphics, 2020, 25(10): 2013−2023. DOI: 10.11834/jig.200253 CrossRef Google Scholar
[6]	夏金泽, 孙浩铭, 胡盛辉, 等. 基于图像信息约束的三维激光点云聚类方法[J]. 光电工程, 2023, 50(2): 220148. DOI: 10.12086/oee.2023.220148 Xia J Z, Sun H M, Hu S H, et al. 3D laser point cloud clustering method based on image information constraints[J]. Opto-Electron Eng, 2023, 50(2): 220148. DOI: 10.12086/oee.2023.220148 CrossRef Google Scholar
[7]	柏宏强, 夏永华, 杨明龙, 等. 基于三维激光点云特征线提取的溶洞多分辨率三维重建方法研究[J]. 激光与光电子学进展, 2020, 57(20): 202802. DOI: 10.3788/LOP57.202802 Bai H Q, Xia Y H, Yang M L, et al. Multi-resolution 3D reconstruction of karst caves based on the feature line extraction of 3D laser point cloud[J]. Laser Optoelectron Prog, 2020, 57(20): 202802. DOI: 10.3788/LOP57.202802 CrossRef Google Scholar
[8]	Maturana D, Scherer S. Voxnet: a 3D convolutional neural network for real-time object recognition[C]//Proceedings of 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2015: 922–928. https://doi.org/10.1109/IROS.2015.7353481. https://doi.org/10.1109/IROS.2015.7353481. " target="_blank">Google Scholar
[9]	Riegler G, Osman Ulusoy A, Geiger A. OctNet: learning deep 3D representations at high resolutions[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition, 2017: 6620–6629. https://doi.org/10.1109/CVPR.2017.701. https://doi.org/10.1109/CVPR.2017.701. " target="_blank">Google Scholar
[10]	肖正涛, 高健, 吴东庆, 等. 一种基于体素网格的三维点云均匀降采样方法[J]. 机械设计与制造, 2023(8): 180−184. DOI: 10.19356/j.cnki.1001-3997.20230310.025 Xiao Z T, Gao J, Wu D Q, et al. A uniform downsampling method for three-dimensional point clouds based on voxel grids[J]. Mach Des Manuf, 2023(8): 180−184. DOI: 10.19356/j.cnki.1001-3997.20230310.025 CrossRef Google Scholar
[11]	Su H, Maji S, Kalogerakis E, et al. Multi-view convolutional neural networks for 3D shape recognition[C]//Proceedings of 2015 IEEE International Conference on Computer Vision, 2015: 945–953. https://doi.org/10.1109/ICCV.2015.114. https://doi.org/10.1109/ICCV.2015.114. " target="_blank">Google Scholar
[12]	Charles R Q, Su H, Mo K C, et al. PointNet: deep learning on point sets for 3D classification and segmentation[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition, 2017: 77–85. https://doi.org/10.1109/CVPR.2017.16 https://doi.org/10.1109/CVPR.2017.16 " target="_blank">Google Scholar
[13]	Qi C R, Yi L, Su H, et al. PointNet++: Deep hierarchical feature learning on point sets in a metric space[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017: 5105–5114. https://doi.org/10.5555/3295222.3295263. https://doi.org/10.5555/3295222.3295263. " target="_blank">Google Scholar
[14]	Simonovsky M, Komodakis N. Dynamic edge-conditioned filters in convolutional neural networks on graphs[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition, 2017: 29–38. https://doi.org/10.1109/CVPR.2017.11. https://doi.org/10.1109/CVPR.2017.11. " target="_blank">Google Scholar
[15]	Wang Y, Sun Y B, Liu Z W, et al. Dynamic graph CNN for learning on point clouds[J]. ACM Trans Graphics, 2019, 38(5): 146. DOI: 10.1145/3326362 CrossRef Google Scholar
[16]	Zhang K G, Hao M, Wang J, et al. Linked dynamic graph CNN: learning on point cloud via linking hierarchical features[Z]. arXiv: 1904.10014, 2019. https://doi.org/10.48550/arXiv.1904.10014. https://doi.org/10.48550/arXiv.1904.10014. " target="_blank">Google Scholar
[17]	刘斌, 樊云超. 基于改进动态图卷积的点云分类模型[J]. 中国科技论文, 2022, 17(11): 1230−1235, 1266. DOI: 10.3969/j.issn.2095-2783.2022.11.009 Liu B, Fan Y C. A point cloud classification model based on improved dynamic graph convolution[J]. China Sci, 2022, 17(11): 1230−1235, 1266. DOI: 10.3969/j.issn.2095-2783.2022.11.009 CrossRef Google Scholar
[18]	Sun Q, Liu H Y, He J, et al. DAGC: employing dual attention and graph convolution for point cloud based place recognition[C]//Proceedings of 2020 International Conference on Multimedia Retrieval, 2020: 224–232. https://doi.org/10.1145/3372278.3390693. https://doi.org/10.1145/3372278.3390693. " target="_blank">Google Scholar
[19]	Guo M H, Cai J X, Liu Z N, et al. PCT: point cloud transformer[J]. Comput Vis Med, 2021, 7(2): 187−199. DOI: 10.1007/s41095-021-0229-5 CrossRef Google Scholar
[20]	Liu H, Tian S H. Deep 3D point cloud classification and segmentation network based on GateNet[J]. Vis Comput, 2023. https://doi.org/10.1007/s00371-023-02826-w. https://doi.org/10.1007/s00371-023-02826-w. " target="_blank">Google Scholar
[21]	Wu Z R, Song S R, Khosla A, et al. 3D shapeNets: a deep representation for volumetric shapes[C]/Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition, 2015: 1912–1920. https://doi.org/10.1109/CVPR.2015.7298801. https://doi.org/10.1109/CVPR.2015.7298801. " target="_blank">Google Scholar
[22]	Uy M A, Pham Q H, Hua B S, et al. Revisiting point cloud classification: a new benchmark dataset and classification model on real-world data[C]//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision, 2019: 1588–1597. https://doi.org/10.1109/ICCV.2019.00167. https://doi.org/10.1109/ICCV.2019.00167. " target="_blank">Google Scholar
[23]	Yi L, Kim V G, Ceylan D, et al. A scalable active framework for region annotation in 3D shape collections[J]. ACM Trans Graphics, 2016, 35(6): 210. DOI: 10.1145/2980179.2980238 CrossRef Google Scholar
[24]	王子璇, 任明武. DST-Pointnet++: 基于Pointnet++改进的点云分类网络[J]. 计算机与数字工程, 2022, 50(11): 2497−2501. DOI: 10.3969/j.issn.1672-9722.2022.11.026 Wang Z X, Ren M W. DST-Pointnet++: A novel point cloud classification network based on pointnet++[J]. Comput Digit Eng, 2022, 50(11): 2497−2501. DOI: 10.3969/j.issn.1672-9722.2022.11.026 CrossRef Google Scholar
[25]	王本杰, 农丽萍, 张文辉, 等. 基于Spider卷积的三维点云分类与分割网络[J]. 计算机应用, 2020, 40(6): 1607−1612. DOI: 10.11772/j.issn.1001-9081.2019101879 Wang B J, Nong L P, Zhang W H, et al. 3D point cloud classification and segmentation network based on Spider convolution[J]. J Comput Appl, 2020, 40(6): 1607−1612. DOI: 10.11772/j.issn.1001-9081.2019101879 CrossRef Google Scholar
[26]	Lin Z H, Huang S Y, Wang Y C F. Convolution in the cloud: learning deformable kernels in 3D graph convolution networks for point cloud analysis[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 1797–1806. https://doi.org/10.1109/CVPR42600.2020.00187. https://doi.org/10.1109/CVPR42600.2020.00187. " target="_blank">Google Scholar
[27]	Chen L F, Zhang Q. DDGCN: graph convolution network based on direction and distance for point cloud learning[J]. Vis Comput, 2023, 39(3): 863−873. DOI: 10.1007/s00371-021-02351-8 CrossRef Google Scholar
[28]	Qiu S, Anwar S, Barnes N. Dense-resolution network for point cloud classification and segmentation[C]//Proceedings of 2021 IEEE Winter Conference on Applications of Computer Vision, 2021: 3812–3821. https://doi.org/10.1109/WACV48630.2021.00386. https://doi.org/10.1109/WACV48630.2021.00386. " target="_blank">Google Scholar
[29]	Wan J, Xie Z, Xu Y Y, et al. DGANet: a dilated graph attention-based network for local feature extraction on 3D point clouds[J]. Remote Sens, 2021, 13(17): 3484. DOI: 10.3390/rs13173484 CrossRef Google Scholar
[30]	张润梅, 程婷, 尹蕾, 等. 一种注意力融合的多尺度点云分类网络[J]. 淮北师范大学学报(自然科学版), 2023, 44(1): 70−75. DOI: 10.3969/j.issn.2096-8248.2023.01.012 Zhang R M, Cheng T, Yin L, et al. Attention fusion and multi-scale point cloud classification network[J]. J Huaibei Normal Univ (Natl Sci), 2023, 44(1): 70−75. DOI: 10.3969/j.issn.2096-8248.2023.01.012 CrossRef Google Scholar
[31]	国玉恩, 任明武. 基于PointConv改进的点云分类网络[J]. 计算机与数字工程, 2022, 50(12): 2737−2740, 2764. DOI: 10.3969/j.issn.1672-9722.2022.12.026 Guo Y E, Ren M W. Improved point cloud classification network based on PointConv[J]. Comput Digit Eng, 2022, 50(12): 2737−2740, 2764. DOI: 10.3969/j.issn.1672-9722.2022.12.026 CrossRef Google Scholar
[32]	Qiu S, Anwar S, Barnes N. Geometric back-projection network for point cloud classification[J]. IEEE Trans Multimedia, 2021, 24: 1943−1955. DOI: 10.1109/TMM.2021.3074240 CrossRef Google Scholar
[33]	Cheng S L, Chen X W, He X W, et al. PRA-Net: point relation-aware network for 3D point cloud analysis[J]. IEEE Trans Image Process, 2021, 30: 4436−4448. DOI: 10.1109/TIP.2021.3072214 CrossRef Google Scholar
[34]	宋巍, 蔡万源, 何盛琪, 等. 结合动态图卷积和空间注意力的点云分类与分割[J]. 中国图象图形学报, 2021, 26(11): 2691−2702. DOI: 10.11834/jig.200550 Song W, Cai W Y, He S Q, et al. Dynamic graph convolution with spatial attention for point cloud classification and segmentation[J]. J Image Graphics, 2021, 26(11): 2691−2702. DOI: 10.11834/jig.200550 CrossRef Google Scholar
[35]	Zhou H R, Feng Y D, Fang M S, et al. Adaptive graph convolution for point cloud analysis[C]//Proceedings of 2021 IEEE/CVF International Conference on Computer Vision, 2021: 4945–4954. https://doi.org/10.1109/ICCV48922.2021.00492. https://doi.org/10.1109/ICCV48922.2021.00492. " target="_blank">Google Scholar

View full references list

Cited By

Cited by

Periodical cited type(3)

1.	赵哲，周维龙，龙欢，周红，梁登辉，柳骏涛，李小宝. 基于遗传算法的激光雷达点云半径滤波. 激光技术. 2025(02): 210-215 .
2.	刘太伟，郁梅，屠仁伟. 基于三维与二维特征融合的无参考点云质量评价. 光电工程. 2025(04): 122-132 . 本站查看
3.	赵德胜，高德东，苏伟鸿，张帅. 基于形状因子与改进分水岭分割的光伏检测算法. 计算机应用. 2024(S2): 343-348 .

Other cited types(3)

Author Information

Author Information
- Tao Zhiyong, xyzmail@126.com On this Site On Google Scholar
  - School of Electronic and Information Engineering, Liaoning Technical University, Huludao, Liaoning 125100, China
- Corresponding author: Li Heng, PaperLH@163.com On this Site On Google Scholar
  PaperLH@163.com
  - School of Electronic and Information Engineering, Liaoning Technical University, Huludao, Liaoning 125100, China
- Dou Miaosen, doumiaosen@163.com On this Site On Google Scholar
  - School of Electronic and Information Engineering, Liaoning Technical University, Huludao, Liaoning 125100, China
- Lin Sen, lin_sen6@126.com On this Site On Google Scholar
  - School of Automation and Electrical Engineering, Shenyang Ligong University, Shenyang, Liaoning 110159, China

Copyright

The copyright belongs to the Institute of Optics and Electronics, Chinese Academy of Sciences, but the article content can be freely downloaded from this website and used for free in academic and research work.

About this Article

About this Article
DOI: 10.12086/oee.2023.230166

Cite this Article

Tao Z Y, Li H, Dou M S, et al. Multi-resolution feature fusion for point cloud classification and segmentation network[J]. Opto-Electron Eng, 2023, 50(10): 230166. DOI: 10.12086/oee.2023.230166

Tao Z Y, Li H, Dou M S, et al. Multi-resolution feature fusion for point cloud classification and segmentation network[J]. Opto-Electron Eng, 2023, 50(10): 230166. DOI: 10.12086/oee.2023.230166

Download Citation
Article History
- Received Date July 06, 2023
- Revised Date September 12, 2023
- Accepted Date September 19, 2023
- Published Date October 24, 2023
Article Metrics

Article Views(2789) PDF Downloads(1188)
Share:

Related Articles
- No-reference point cloud quality assessment based on fusion of 3D and 2D features
  
  Liu Taiwei, Yu Mei, Tu Renwei
  
  Opto-Electronic Engineering, DOI: 10.12086/oee.2025.250001
- Remote sensing image road extraction by integrating ResNeSt and multi-scale feature fusion
  
  Hao Ming, Bai He, Xu Tingting
  
  Opto-Electronic Engineering, DOI: 10.12086/oee.2025.240236
- Adaptive feature fusion cascade Transformer retinal vessel segmentation algorithm
  
  Liang Liming, Lu Baohe, Long Pengwei, Yang Yuan
  
  Opto-Electronic Engineering, DOI: 10.12086/oee.2023.230161
- Point cloud-image data fusion for road segmentation
  
  Zhang Ying, Huang Yingping, Guo Zhiyang, Zhang Chong
  
  Opto-Electronic Engineering, DOI: 10.12086/oee.2021.210340
- Fusing point cloud with image for object detection using convolutional neural networks
  
  Zhang Jiesong, Huang Yingping, Zhang Rui
  
  Opto-Electronic Engineering, DOI: 10.12086/oee.2021.200418
- Light-field image super-resolution based on multi-scale feature fusion
  
  Zhao Yuanyuan, Shi Shengxian
  
  Opto-Electronic Engineering, DOI: 10.12086/oee.2020.200007
- Image super-resolution reconstruction based on multi-scale feature loss function
  
  Xu Liang, Fu Randi, Jin Wei, Tang Biao, Wang Shangli
  
  Opto-Electronic Engineering, DOI: 10.12086/oee.2019.180419
- Ground segmentation from 3D point cloud using features of scanning line segments
  
  Cheng Ziyang, Ren Guoquan, Zhang Yin
  
  Opto-Electronic Engineering, DOI: 10.12086/oee.2019.180268
- Background modeling method based on multi-feature fusion
  
  Guo Zhicheng, Dang Jianwu, Wang Yangping, Jin Jing
  
  Opto-Electronic Engineering, DOI: 10.12086/oee.2018.180206
- Image super-resolution reconstruction by fusing feature classification and independent dictionary training
  
  Wang Ronggui, Wang Qinghui, Yang Juan, Hu Min
  
  Opto-Electronic Engineering, DOI: 10.12086/oee.2018.170542

Article Contents
Figures/Tables
References

View in article Downloads

Full-Size Img PowerPoint
View in article Downloads

Full-Size Img PowerPoint
View in article Downloads

Full-Size Img PowerPoint
View in article Downloads

Full-Size Img PowerPoint
View in article Downloads

Full-Size Img PowerPoint
View in article Downloads

Full-Size Img PowerPoint
View in article Downloads

Full-Size Img PowerPoint
View in article Downloads

Full-Size Img PowerPoint

参数项	分类网络	分割网络
输入点数	1024	2048
多分辨率点云点数	[896,768,640,512]	[896,768,640]
图卷积分支k取值	20	20
训练周期	250	300
优化器	SGD	SGD
训练批次	32	32
测试批次	16	16
初始学习速率	0.1	0.003

View in article Downloads

方法	输入	点数/10³	mAcc/%	OA/%
VoxNet^[8]	体素	-	83.0	85.9
MVCNN^[11]	多视图	-	-	90.1
PointNet^[12]	坐标	1	86.0	89.2
PointNet++^[13]	坐标+法线	5	-	91.9
文献[24]	坐标+法线	1	89.8	91.6
文献[25]	坐标+法线	1	-	93.0
3D-GCN^[26]	坐标	1	-	92.1
DGCNN^[15]	坐标	1	90.2	92.9
LDGCNN^[16]	坐标	1	90.3	92.9
DDGCN^[27]	坐标	1	90.4	92.7
DRNet^[28]	坐标	1	-	93.1
DGANet^[29]	坐标	1	89.4	92.3
PCT^[19]	坐标	1	-	93.2
AFM-Net^[30]	坐标	1	89.4	92.85
文献[31]	坐标	1	89.02	92.5
Our	坐标	1	91.2	93.4

View in article Downloads

方法输入 mAcc/% OA/%

PointNet^[12] 坐标 63.4 68.2
PointNet++^[13] 坐标 75.4 77.9
DGCNN^[15] 坐标 73.6 78.1
DRNet^[28] 坐标 78.0 80.3
GBNet^[32] 坐标 77.8 80.5
PRANet^[33] 坐标 79.1 82.1
Ours 坐标 81.7 83.3

View in article Downloads

方法	PointNet^[12]	PointNet++^[13]	文献[25]	3D-GCN^[26]	LDGCNN^[16]	DGANet^[29]	DGCSA^[34]	DGCNN^[15]	本文
飞机	83.4	82.4	83.8	83.1	84.0	84.6	84.2	84.0	83.6
包	78.7	79.0	77.5	84.0	83.0	85.7	73.3	83.4	83.4
帐篷	82.5	87.7	87.9	86.6	84.9	87.8	82.3	86.7	88.4↑
车	74.9	77.3	78.7	77.5	78.4	78.5	77.7	77.8	78.4↑
椅子	89.6	90.8	90.8	90.3	90.6	91.0	91.0	90.6	89.7
耳机	73.0	71.8	77.3	74.1	74.4	77.3	75.3	74.7	80.5↑
吉他	91.5	91.0	91.8	90.9	91.0	91.2	91.2	91.2	91.8↑
刀	85.9	85.9	87.9	86.4	88.1	87.9	88.6	87.5	88.6↑
台灯	80.8	83.7	84.2	83.8	83.4	82.4	85.3	82.8	81.6
手提电脑	95.3	95.3	95.9	95.6	95.8	95.8	95.9	95.7	95.8↑
摩托	65.2	71.6	71.8	66.8	67.4	67.8	58.9	66.3	69.6↑
马克杯	93.0	94.1	95.1	94.8	94.9	94.2	94.3	94.9	94.4
手枪	81.2	81.3	80.9	81.3	82.3	81.1	81.8	81.1	83.7↑
火箭	57.9	58.7	59.6	59.6	59.2	59.7	56.9	63.5	62.5
滑板	72.8	76.4	76.6	75.7	76.0	75.7	75.4	74.5	82.0↑
桌子	80.6	82.6	82.4	82.8	81.9	82.0	82.7	82.6	83.0↑
mIoU	83.7	85.1	85.4	85.1	85.1	85.2	85.3	85.2	85.4↑

View in article Downloads

k OA(%) 用多分辨率分支补偿后OA/% 提升/%

5 20.7 35.1 +14.4
10 85.4 88.3 +2.9
15 91.9 92.1 +0.2
20 92.5 93.4 +0.9
25 92.1 92.3 +0.2

View in article Downloads
实验 GCN分支 M-R分支融合 mAcc/% OA/%

1 √ × × 89.9 92.5
2 × √ × 84.0 89.1
3 √ √ × 89.9 92.6
4 √ √ √ 91.2 93.4

View in article Downloads
不同分辨率的点云 mAcc/% OA/%

[512,384,256,128] 90.4 92.7
[640,512,384,256] 90.6 92.8
[768,640,512,384] 90.9 93.0
[896,768,640,512] 91.2 93.4

View in article Downloads
噪声水平下降程度
3D-GCN AdaptConv DGCNN Ours

0.02 0.7↓ 1.8↓ 1.4↓ 0.9↓
0.04 2.2↓ 2.2↓ 2.2↓ 1.8↓
0.06 4.6↓ 3.3↓ 3.2↓ 3.1↓
0.08 8.4↓ 6.5↓ 5.7↓ 6.4↓
0.1 14.9↓ 10.8↓ 13.1↓ 11.7↓

View in article Downloads
模块数量 mAcc/% OA/% 每轮训练时间/s 模型参数量/M

3 89.7 92.4 63 2.8
4 91.2 93.4 139 3.6
5 90.6 93.1 323 4.8

View in article Downloads

[1]	张昕怡, 陈茂霖, 刘祥江, 等. 顾及点密度与未知角分辨率的地面点云分类[J]. 激光技术, 2023, 47(1): 59−66. DOI: 10.7510/jgjs.issn.1001-3806.2023.01.009 Zhang X Y, Chen M L, Liu X J, et al. Classification of terrestrial point cloud considering point density and unknown angular resolution[J]. Laser Technol, 2023, 47(1): 59−66. DOI: 10.7510/jgjs.issn.1001-3806.2023.01.009 CrossRef Google Scholar
[2]	李佳男, 王泽, 许廷发. 基于点云数据的三维目标检测技术研究进展[J]. 光学学报, 2023, 43(15): 1515001. DOI: 10.3788/AOS230745 Li J N, Wang Z, Xu T F. Three-dimensional object detection technology based on point cloud data[J]. Acta Opt Sin, 2023, 43(15): 1515001. DOI: 10.3788/AOS230745 CrossRef Google Scholar
[3]	陆康亮, 薛俊, 陶重犇. 融合空间掩膜预测与点云投影的多目标跟踪[J]. 光电工程, 2022, 49(9): 220024. DOI: 10.12086/oee.2022.220024 Lu K L, Xue J, Tao C B. Multi target tracking based on spatial mask prediction and point cloud projection[J]. Opto-Electron Eng, 2022, 49(9): 220024. DOI: 10.12086/oee.2022.220024 CrossRef Google Scholar
[4]	陈仲生, 李潮林, 左旺, 等. 双重下采样增强的点云改进配准算法研究[J]. 汽车工程, 2023, 45(4): 572−578. DOI: 10.19562/j.chinasae.qcgc.2023.04.005 Chen Z S, Li C L, Zuo W, et al. Study on improved point cloud registration algorithm enhanced by double down-sampling[J]. Automot Eng, 2023, 45(4): 572−578. DOI: 10.19562/j.chinasae.qcgc.2023.04.005 CrossRef Google Scholar
[5]	李美佳, 于泽宽, 刘晓, 等. 点云算法在医学领域的研究进展[J]. 中国图象图形学报, 2020, 25(10): 2013−2023. DOI: 10.11834/jig.200253 Li M J, Yu Z K, Liu X, et al. Progress of point cloud algorithm in medical field[J]. J Image Graphics, 2020, 25(10): 2013−2023. DOI: 10.11834/jig.200253 CrossRef Google Scholar
[6]	夏金泽, 孙浩铭, 胡盛辉, 等. 基于图像信息约束的三维激光点云聚类方法[J]. 光电工程, 2023, 50(2): 220148. DOI: 10.12086/oee.2023.220148 Xia J Z, Sun H M, Hu S H, et al. 3D laser point cloud clustering method based on image information constraints[J]. Opto-Electron Eng, 2023, 50(2): 220148. DOI: 10.12086/oee.2023.220148 CrossRef Google Scholar
[7]	柏宏强, 夏永华, 杨明龙, 等. 基于三维激光点云特征线提取的溶洞多分辨率三维重建方法研究[J]. 激光与光电子学进展, 2020, 57(20): 202802. DOI: 10.3788/LOP57.202802 Bai H Q, Xia Y H, Yang M L, et al. Multi-resolution 3D reconstruction of karst caves based on the feature line extraction of 3D laser point cloud[J]. Laser Optoelectron Prog, 2020, 57(20): 202802. DOI: 10.3788/LOP57.202802 CrossRef Google Scholar
[8]	Maturana D, Scherer S. Voxnet: a 3D convolutional neural network for real-time object recognition[C]//Proceedings of 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2015: 922–928. https://doi.org/10.1109/IROS.2015.7353481. https://doi.org/10.1109/IROS.2015.7353481. " target="_blank">Google Scholar
[9]	Riegler G, Osman Ulusoy A, Geiger A. OctNet: learning deep 3D representations at high resolutions[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition, 2017: 6620–6629. https://doi.org/10.1109/CVPR.2017.701. https://doi.org/10.1109/CVPR.2017.701. " target="_blank">Google Scholar
[10]	肖正涛, 高健, 吴东庆, 等. 一种基于体素网格的三维点云均匀降采样方法[J]. 机械设计与制造, 2023(8): 180−184. DOI: 10.19356/j.cnki.1001-3997.20230310.025 Xiao Z T, Gao J, Wu D Q, et al. A uniform downsampling method for three-dimensional point clouds based on voxel grids[J]. Mach Des Manuf, 2023(8): 180−184. DOI: 10.19356/j.cnki.1001-3997.20230310.025 CrossRef Google Scholar
[11]	Su H, Maji S, Kalogerakis E, et al. Multi-view convolutional neural networks for 3D shape recognition[C]//Proceedings of 2015 IEEE International Conference on Computer Vision, 2015: 945–953. https://doi.org/10.1109/ICCV.2015.114. https://doi.org/10.1109/ICCV.2015.114. " target="_blank">Google Scholar
[12]	Charles R Q, Su H, Mo K C, et al. PointNet: deep learning on point sets for 3D classification and segmentation[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition, 2017: 77–85. https://doi.org/10.1109/CVPR.2017.16 https://doi.org/10.1109/CVPR.2017.16 " target="_blank">Google Scholar
[13]	Qi C R, Yi L, Su H, et al. PointNet++: Deep hierarchical feature learning on point sets in a metric space[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017: 5105–5114. https://doi.org/10.5555/3295222.3295263. https://doi.org/10.5555/3295222.3295263. " target="_blank">Google Scholar
[14]	Simonovsky M, Komodakis N. Dynamic edge-conditioned filters in convolutional neural networks on graphs[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition, 2017: 29–38. https://doi.org/10.1109/CVPR.2017.11. https://doi.org/10.1109/CVPR.2017.11. " target="_blank">Google Scholar
[15]	Wang Y, Sun Y B, Liu Z W, et al. Dynamic graph CNN for learning on point clouds[J]. ACM Trans Graphics, 2019, 38(5): 146. DOI: 10.1145/3326362 CrossRef Google Scholar
[16]	Zhang K G, Hao M, Wang J, et al. Linked dynamic graph CNN: learning on point cloud via linking hierarchical features[Z]. arXiv: 1904.10014, 2019. https://doi.org/10.48550/arXiv.1904.10014. https://doi.org/10.48550/arXiv.1904.10014. " target="_blank">Google Scholar
[17]	刘斌, 樊云超. 基于改进动态图卷积的点云分类模型[J]. 中国科技论文, 2022, 17(11): 1230−1235, 1266. DOI: 10.3969/j.issn.2095-2783.2022.11.009 Liu B, Fan Y C. A point cloud classification model based on improved dynamic graph convolution[J]. China Sci, 2022, 17(11): 1230−1235, 1266. DOI: 10.3969/j.issn.2095-2783.2022.11.009 CrossRef Google Scholar
[18]	Sun Q, Liu H Y, He J, et al. DAGC: employing dual attention and graph convolution for point cloud based place recognition[C]//Proceedings of 2020 International Conference on Multimedia Retrieval, 2020: 224–232. https://doi.org/10.1145/3372278.3390693. https://doi.org/10.1145/3372278.3390693. " target="_blank">Google Scholar
[19]	Guo M H, Cai J X, Liu Z N, et al. PCT: point cloud transformer[J]. Comput Vis Med, 2021, 7(2): 187−199. DOI: 10.1007/s41095-021-0229-5 CrossRef Google Scholar
[20]	Liu H, Tian S H. Deep 3D point cloud classification and segmentation network based on GateNet[J]. Vis Comput, 2023. https://doi.org/10.1007/s00371-023-02826-w. https://doi.org/10.1007/s00371-023-02826-w. " target="_blank">Google Scholar
[21]	Wu Z R, Song S R, Khosla A, et al. 3D shapeNets: a deep representation for volumetric shapes[C]/Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition, 2015: 1912–1920. https://doi.org/10.1109/CVPR.2015.7298801. https://doi.org/10.1109/CVPR.2015.7298801. " target="_blank">Google Scholar
[22]	Uy M A, Pham Q H, Hua B S, et al. Revisiting point cloud classification: a new benchmark dataset and classification model on real-world data[C]//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision, 2019: 1588–1597. https://doi.org/10.1109/ICCV.2019.00167. https://doi.org/10.1109/ICCV.2019.00167. " target="_blank">Google Scholar
[23]	Yi L, Kim V G, Ceylan D, et al. A scalable active framework for region annotation in 3D shape collections[J]. ACM Trans Graphics, 2016, 35(6): 210. DOI: 10.1145/2980179.2980238 CrossRef Google Scholar
[24]	王子璇, 任明武. DST-Pointnet++: 基于Pointnet++改进的点云分类网络[J]. 计算机与数字工程, 2022, 50(11): 2497−2501. DOI: 10.3969/j.issn.1672-9722.2022.11.026 Wang Z X, Ren M W. DST-Pointnet++: A novel point cloud classification network based on pointnet++[J]. Comput Digit Eng, 2022, 50(11): 2497−2501. DOI: 10.3969/j.issn.1672-9722.2022.11.026 CrossRef Google Scholar
[25]	王本杰, 农丽萍, 张文辉, 等. 基于Spider卷积的三维点云分类与分割网络[J]. 计算机应用, 2020, 40(6): 1607−1612. DOI: 10.11772/j.issn.1001-9081.2019101879 Wang B J, Nong L P, Zhang W H, et al. 3D point cloud classification and segmentation network based on Spider convolution[J]. J Comput Appl, 2020, 40(6): 1607−1612. DOI: 10.11772/j.issn.1001-9081.2019101879 CrossRef Google Scholar
[26]	Lin Z H, Huang S Y, Wang Y C F. Convolution in the cloud: learning deformable kernels in 3D graph convolution networks for point cloud analysis[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 1797–1806. https://doi.org/10.1109/CVPR42600.2020.00187. https://doi.org/10.1109/CVPR42600.2020.00187. " target="_blank">Google Scholar
[27]	Chen L F, Zhang Q. DDGCN: graph convolution network based on direction and distance for point cloud learning[J]. Vis Comput, 2023, 39(3): 863−873. DOI: 10.1007/s00371-021-02351-8 CrossRef Google Scholar
[28]	Qiu S, Anwar S, Barnes N. Dense-resolution network for point cloud classification and segmentation[C]//Proceedings of 2021 IEEE Winter Conference on Applications of Computer Vision, 2021: 3812–3821. https://doi.org/10.1109/WACV48630.2021.00386. https://doi.org/10.1109/WACV48630.2021.00386. " target="_blank">Google Scholar
[29]	Wan J, Xie Z, Xu Y Y, et al. DGANet: a dilated graph attention-based network for local feature extraction on 3D point clouds[J]. Remote Sens, 2021, 13(17): 3484. DOI: 10.3390/rs13173484 CrossRef Google Scholar
[30]	张润梅, 程婷, 尹蕾, 等. 一种注意力融合的多尺度点云分类网络[J]. 淮北师范大学学报(自然科学版), 2023, 44(1): 70−75. DOI: 10.3969/j.issn.2096-8248.2023.01.012 Zhang R M, Cheng T, Yin L, et al. Attention fusion and multi-scale point cloud classification network[J]. J Huaibei Normal Univ (Natl Sci), 2023, 44(1): 70−75. DOI: 10.3969/j.issn.2096-8248.2023.01.012 CrossRef Google Scholar
[31]	国玉恩, 任明武. 基于PointConv改进的点云分类网络[J]. 计算机与数字工程, 2022, 50(12): 2737−2740, 2764. DOI: 10.3969/j.issn.1672-9722.2022.12.026 Guo Y E, Ren M W. Improved point cloud classification network based on PointConv[J]. Comput Digit Eng, 2022, 50(12): 2737−2740, 2764. DOI: 10.3969/j.issn.1672-9722.2022.12.026 CrossRef Google Scholar
[32]	Qiu S, Anwar S, Barnes N. Geometric back-projection network for point cloud classification[J]. IEEE Trans Multimedia, 2021, 24: 1943−1955. DOI: 10.1109/TMM.2021.3074240 CrossRef Google Scholar
[33]	Cheng S L, Chen X W, He X W, et al. PRA-Net: point relation-aware network for 3D point cloud analysis[J]. IEEE Trans Image Process, 2021, 30: 4436−4448. DOI: 10.1109/TIP.2021.3072214 CrossRef Google Scholar
[34]	宋巍, 蔡万源, 何盛琪, 等. 结合动态图卷积和空间注意力的点云分类与分割[J]. 中国图象图形学报, 2021, 26(11): 2691−2702. DOI: 10.11834/jig.200550 Song W, Cai W Y, He S Q, et al. Dynamic graph convolution with spatial attention for point cloud classification and segmentation[J]. J Image Graphics, 2021, 26(11): 2691−2702. DOI: 10.11834/jig.200550 CrossRef Google Scholar
[35]	Zhou H R, Feng Y D, Fang M S, et al. Adaptive graph convolution for point cloud analysis[C]//Proceedings of 2021 IEEE/CVF International Conference on Computer Vision, 2021: 4945–4954. https://doi.org/10.1109/ICCV48922.2021.00492. https://doi.org/10.1109/ICCV48922.2021.00492. " target="_blank">Google Scholar

No-reference point cloud quality assessment based on fusion of 3D and 2D features

Liu Taiwei, Yu Mei, Tu Renwei

Opto-Electronic Engineering, DOI: 10.12086/oee.2025.250001
Remote sensing image road extraction by integrating ResNeSt and multi-scale feature fusion

Hao Ming, Bai He, Xu Tingting

Opto-Electronic Engineering, DOI: 10.12086/oee.2025.240236
Adaptive feature fusion cascade Transformer retinal vessel segmentation algorithm

Liang Liming, Lu Baohe, Long Pengwei, Yang Yuan

Opto-Electronic Engineering, DOI: 10.12086/oee.2023.230161
Point cloud-image data fusion for road segmentation

Zhang Ying, Huang Yingping, Guo Zhiyang, Zhang Chong

Opto-Electronic Engineering, DOI: 10.12086/oee.2021.210340
Fusing point cloud with image for object detection using convolutional neural networks

Zhang Jiesong, Huang Yingping, Zhang Rui

Opto-Electronic Engineering, DOI: 10.12086/oee.2021.200418
Light-field image super-resolution based on multi-scale feature fusion

Zhao Yuanyuan, Shi Shengxian

Opto-Electronic Engineering, DOI: 10.12086/oee.2020.200007
Image super-resolution reconstruction based on multi-scale feature loss function

Xu Liang, Fu Randi, Jin Wei, Tang Biao, Wang Shangli

Opto-Electronic Engineering, DOI: 10.12086/oee.2019.180419
Ground segmentation from 3D point cloud using features of scanning line segments

Cheng Ziyang, Ren Guoquan, Zhang Yin

Opto-Electronic Engineering, DOI: 10.12086/oee.2019.180268
Background modeling method based on multi-feature fusion

Guo Zhicheng, Dang Jianwu, Wang Yangping, Jin Jing

Opto-Electronic Engineering, DOI: 10.12086/oee.2018.180206
Image super-resolution reconstruction by fusing feature classification and independent dictionary training

Wang Ronggui, Wang Qinghui, Yang Juan, Hu Min

Opto-Electronic Engineering, DOI: 10.12086/oee.2018.170542

Multi-resolution feature fusion for point cloud classification and segmentation network

Abstract

Keywords

1. 引 言

2. 原理与方法

2.1 网络整体框架

2.2 图卷积分支

2.3 多分辨率分支

2.4 特征融合模块

3. 实验结果与分析

3.1 实验环境配置

3.2 基于ModelNet40数据集的点云分类

3.2.1 数据集描述

3.2.2 分类结果与分析

3.3 基于ScanObjectNN数据集的点云分类

3.3.1 数据集描述

3.3.2 分类结果与分析

3.4 基于ShapeNet Part数据集的点云分割

3.4.1 数据集描述

3.4.2 分割结果与分析

4. 消融实验

4.1 图卷积分支中k取值对性能的影响

4.2 多分辨率图卷积模块的消融实验

4.3 多分辨率点云规模的选取

4.4 噪声鲁棒性测试

4.5 特征提取模块数量消融实验

5. 结 论

利益冲突

References

Cited by

Periodical cited type(3)

Other cited types(3)

Author Information

Tao Zhiyong, xyzmail@126.com On this SiteOn Google Scholar

Corresponding author: Li Heng, PaperLH@163.com On this SiteOn Google Scholar

Dou Miaosen, doumiaosen@163.com On this SiteOn Google Scholar

Lin Sen, lin_sen6@126.com On this SiteOn Google Scholar

Copyright

About this Article

Cite this Article

Article History

Article Metrics

Related Articles

Links

Related Articles

Catalog

Lin Sen, lin_sen6@126.com

Manuscript Submission

More Content

Legal & Privacy

Export File

Citation

Format

Content

WeChat Qrcode

1. 引　言

5. 结　论

Tao Zhiyong, xyzmail@126.com On this Site On Google Scholar

Corresponding author: Li Heng, PaperLH@163.com On this Site On Google Scholar

Dou Miaosen, doumiaosen@163.com On this Site On Google Scholar

Lin Sen, lin_sen6@126.com On this Site On Google Scholar