Calculation Method of RGBD Scene Flow Combining Gaussian Mixture Model and Multi-channel Bilateral Filtering
WANG Zige1,2, LI Yingying1,2, GE Liyue1,3, CHEN Zhen1,2,4, ZHANG Congxuan1,2,4
1. Key Laboratory of Jiangxi Province for Image Processing and Pattern Recognition, Nanchang Hangkong University, Nanchang 330063; 2. School of Measuring and Optical Engineering, Nanchang Hangkong University, Nanchang 330063; 3. School of Information Engineering, Nanchang Hangkong University, Nanchang 330063; 4. Key Laboratory of Non-destructive Testing Technology of Ministry of Education, Nanchang Hangkong University, Nanchang 330063
Abstract To improve the computational accuracy and robustness of existing RGBD scene flow calculation methods under complex motion scenarios, such as large displacement and motion occlusion, a calculation method of RGBD scene flow combining Gaussian mixture model and multi-channel bilateral filtering is proposed. Firstly, a Gaussian mixture-based optical flow clustering segmentation model is constructed to extract target motion information from optical flow and optimize the results of depth map segmentation layer by layer. Consequently, high-confidence depth motion hierarchical segmentation information is obtained. Then, the RGBD scene flow estimation model combining the Gaussian mixture model and multi-channel bilateral filtering is established by introducing the multi-channel bilateral filtering optimization to overcome the edge-blurring problem of the scene flow computation. Finally, experiments on Middlebury and MPI-Sintel datasets demonstrate that the proposed method exhits higher accuracy and robustness in complex motion scenarios such as large displacements and motion occlusions, particularly in edge-preserving.
Fund:National Key Research and Development Program of China(No.2020YFC2003800), National Natural Science Foundation of China(No.62222206,62272209,61866026,61772255,61866025), Key Program of Natural Science Foundation of Jiangxi Province(No.20202ACB214007), Technology Innovation Guidance Program of Jiangxi Province(No.20212AEI91005), Advantage Subject Team Project of Jiangxi Province(No.20165BCB19007), Science and Technology Research Project of Education Department of Jiangxi Province(No.GJJ210910), Key Laboratory of Image Processing and Pattern Recognition Open Fund of Jiangxi Province(No.ET202104413), Innovation Fund Designated for Graduate Students of Jiangxi Province(No.YC2021-S689)
Corresponding Authors:ZHANG Congxuan, Ph.D., professor. His research inte-rests include image processing and computer vision.
About author:: WANG Zige, master student. Her research interests include computer vision. LI Yingying, master student. Her research interests include computer vision. GE Liyue, Ph.D. candidate, assistant experimentalist. His research interests include machine vision and intelligent perception. CHEN Zhen, Ph.D., professor. His research interests include image processing and computer vision.
WANG Zige,LI Yingying,GE Liyue等. Calculation Method of RGBD Scene Flow Combining Gaussian Mixture Model and Multi-channel Bilateral Filtering[J]. Pattern Recognition and Artificial Intelligence, 2023, 36(3): 225-241.
[1] 葛利跃,邓士心,龚洁,等.基于运动优化语义分割的变分光流计算方法.模式识别与人工智能, 2021, 34(7): 631-645. (GE L Y, DENG S X, GONG J, et al. Variational Optical Flow Computation Method Based on Motion Optimization Semantic Segmentation. Pattern Recognition and Artificial Intelligence, 2021, 34(7): 631-645.) [2] SCHUSTER R, WASENMÜLLER O, UNGER C, et al. SceneFlowFields++: Multi-frame Matching, Visibility Prediction, and Robust Interpolation for Scene Flow Estimation. International Journal of Computer Vision, 2020, 128(2): 527-546. [3] PONTES J K, HAYS J, LUCEY S. Scene Flow from Point Clouds with or Without Learning // Proc of the International Conference on 3D Vision. Washington, USA: IEEE, 2020: 261-270. [4] SAXENA R, SCHUSTER R, WASENMULLER O, et al. PWOC-3D: Deep Occlusion-Aware End-to-End Scene Flow Estimation // Proc of the IEEE Intelligent Vehicles Symposium. Washington, USA: IEEE, 2019: 324-331. [5] OUYANG B J, RAVIV D. Occlusion Guided Self-Supervised Scene Flow Estimation on 3D Point Clouds // Proc of the International Conference on 3D Vision. Washington, USA: IEEE, 2021: 782-791. [6] VEDULA S, RANDER P, COLLINS R, et al. Three-Dimensional Scene Flow. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2005, 27(3): 475-480. [7] 陈震,马龙,张聪炫,等.基于语义分割的双目场景流估计.电子学报, 2020, 48(4): 631-636. (CHEN Z, MA L, ZHANG C X, et al. Binocular Scene Flow Estimation Based on Semantic Segmentation. Acta Electronica Sinica, 2020, 48(4): 631-636.) [8] WEDEL A, RABE C, VAUDREY T, et al. Efficient Dense Scene Flow from Sparse or Dense Stereo Data // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2008: 739-751. [9] RABE C, MÜLLER T, WEDEL A, et al. Dense, Robust, and Accurate Motion Field Estimation from Stereo Image Sequences in Real-Time // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2010: 582-595. [10] BASHA T, MOSES Y, KIRYATI N. Multi-view Scene Flow Estimation: A View Centered Variational Approach. International Journal of Computer Vision, 2013, 101(1): 6-21. [11] MENZE M, GEIGER A. Object Scene Flow for Autonomous Vehicles // Proc of the IEEE Conference on Computer Vision and Pa-ttern Recognition. Washington, USA: IEEE, 2015: 3061-3070. [12] VOGEL C, SCHINDLER K, ROTH S. Piecewise Rigid Scene Flow // Proc of the IEEE Conference on Computer Vision. Wa-shington, USA: IEEE, 2013: 1377-1384. [13] SCHUSTER R, WASENMULLER O, KUSCHK G, et al. SceneFlowFields: Dense Interpolation of Sparse Scene Flow Correspondences // Proc of the IEEE Winter Conference on Applications of Computer Vision. Washington, USA: IEEE, 2018: 1056-1065. [14] ZHOU Z K, FAN X N, SHI P F, et al. R-MSFM: Recurrent Multi-scale Feature Modulation for Monocular Depth Estimating // Proc of the IEEE/CVF International Conference on Computer Vision. Washington, USA: IEEE, 2021: 12757-12766. [15] GOTTFRIED J M, FEHR J, GARBE C S. Computing Range Flow from Multi-modal Kinect Data // Proc of the International Sympo-sium on Visual Computing. Berlin, Germany: Springer, 2011: 758-767. [16] QUIROGA J, DEVERNAY F, CROWLEY J. Local/Global Scene Flow Estimation // Proc of the IEEE International Conference on Image Processing. Washington, USA: IEEE, 2013: 3850-3854. [17] HERBST E, REN X F, FOX D. RGB-D Flow: Dense 3-D Motion Estimation Using Color and Depth // Proc of the IEEE Internatio-nal Conference on Robotics and Automation. Washington, USA: IEEE, 2013: 2276-2282. [18] WANG Y C, ZHANG J, LIU Z C, et al. Handling Occlusion and Large Displacement Through Improved RGB-D Scene Flow Estimation. IEEE Transactions on Circuits and Systems for Video Te-chnology, 2016, 26(7): 1265-1278. [19] SUN D Q, SUDDERTH E B, PFISTER H. Layered RGBD Scene Flow Estimation // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2015: 548-556. [20] MAYER N, IIG E, HÄUSSER P, et al. A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 4040-4048. [21] EIGEN D, PUHRSCH C, FERGUS R.Depth Map Prediction from a Single Image Using a Multi-scale Deep Network // Proc of the 27th International Conference on Neural Information Processing Systems. Cambridge, USA: MIT Press, 2014, II: 2366-2374. [22] YANG G S, RAMANAN D. Upgrading Optical Flow to 3D Scene Flow Through Optical Expansion // Proc of the IEEE/CVF Confe-rence on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2020: 1331-1340. [23] QUIROGA J, BROX T, DEVERNAY F, et al. Dense Semi-Rigid Scene Flow Estimation from RGBD Images // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2014: 567-582. [24] ZHAI M L, XIANG X Z, LÜ N, et al. Optical Flow and Scene Flow Estimation: A Survey. Pattern Recognition, 2021, 114. DOI: 10.1016/j.patcog.2021.107861. [25] SUN D Q, ROTH S, BLACK M J. A Quantitative Analysis of Cu-rrent Practices in Optical Flow Estimation and the Principles Behind Them. International Journal of Computer Vision, 2014, 106(2): 115-137. [26] ZHANG C X, CHEN Z, WANG M R, et al. Robust Non-local TV-L1 Optical Flow Estimation with Occlusion Detection. IEEE Transactions on Image Processing, 2017, 26(8): 4055-4067. [27] DONG C, WANG Z S, HAN J M, et al. A Non-local Propagation Filtering Scheme for Edge-Preserving in Variational Optical Flow Computation. Signal Processing: Image Communication, 2021, 93. DOI: 10.1016/j.image.2021.116143. [28] ZHU Y, LAN Z Z, NEWSAM S, et al. Guided Optical Flow Learning[C/OL].[2022-11-23]. https://arxiv.org/pdf/1702.02295.pdf. [29] LI X X, LIU Y J, JIN H Y, et al. Automatic Layered RGB-D Scene Flow Estimation with Optical Flow Field Constraint. IET Image Processing, 2020, 14(8). DOI: 10.1049/iet-ipr.2020.0230. [30] BROX T, MALIK J. Large Displacement Optical Flow: Descriptor Matching in Variational Motion Estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, 33(3): 500-513. [31] 张聪炫,裴刘继,陈震,等.FRFCM聚类与深度优化的RGBD场景流计算.电子学报, 2020, 48(7): 1380-1386. (ZHANG C X, PEI L J, CHEN Z, et al. RGBD Scene Flow Estimation Based on FRFCM Clustering and Depth Optimization. Acta Electronica Sinica, 2020, 48(7): 1380-1386.)
2023, Vol. 36 
