Stereo Matching Algorithm Based on Control Points, RGB Vector Difference and Gradient Census Transform
WANG Sen1,2, WEI Hui1,2, MENG Lingjiang1,2
1. Laboratory of Algorithms for Cognitive Modeling, School of Computer Science, Fudan University, Shanghai 200438; 2. Shanghai Key Laboratory of Data Science, School of Computer Science, Fudan University, Shanghai 200438
Abstract Aiming at the low matching precision of weak texture region and boundary in traditional stereo matching algorithms, a stereo matching algorithm based on control points, RGB vector difference and gradient census transform is proposed. Firstly, a row matching algorithm based on dynamic time warping(DTW) is exploited to find the optimal matching path. The path is distorted and aligned to select matching control points. Then, RGB vector difference cost is combined with gradient Census transform cost as the matching cost of non-control points. The robustness of pixels is enhanced by gradient Census transform, and the three-dimensional color information of the image is retained by the RGB vector. Thus the higher matching precision is achieved. The combination of the matching cost of control points and non-control points is regarded as the initial matching cost. The RGB vector difference is utilized to acquire the adaptive window of different texture areas. The cost aggregation is calculated by the algorithm in a horizontal and vertical directions of an adaptive window. The cost aggregation is employed to optimize the initial cost. Multi-step optimization is employed to reduce the disparity error rate.Finally, the proposed algorithm is validated on Middlebury dataset for different regions and the disparity is calculated in the real scene of robot. The calculated disparity is verified for three-dimensional reconstruction based on the principle of 3D imaging. Both theoretical analysis and experimental results show that the proposed algorithm reduces the matching error rate of weak texture region and boundary significantly.
Fund:National Natural Science Foundation of China(No.61771146)
Corresponding Authors:
WEI Hui, Ph.D., professor. His research interests include cognitive science and artificial intelligence.
About author:: WANG Sen, master student. Her research interests include stereo vision. MENG Lingjiang, Ph.D. candidate. His research interests include stereo vision.
WANG Sen,WEI Hui,MENG Lingjiang. Stereo Matching Algorithm Based on Control Points, RGB Vector Difference and Gradient Census Transform[J]. Pattern Recognition and Artificial Intelligence, 2022, 35(1): 37-50.
[1] WEI H, XU C, JIN Z F.Binocular Matching Model Based on Hie-rarchical V1 and V2 Receptive Fields with Color, Orientation, and Region Feature Information. IEEE Transactions on Biomedical Engineering, 2020, 67(11): 3141-3150. [2] GEIGER A, LENZ P, URTASUN R.Are We Ready for Autonomous Driving? the KITTI Vision Benchmark Suite // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2012: 3354-3361. [3] HAMZAH R A, KADIN A F, HAMID M S, et al. Improvement of Stereo Matching Algorithm for 3D Surface Reconstruction. Signal Processing: Image Communication, 2018, 65: 165-172. [4] ZHAI G D, ZHANG W T, HU W Y, et al. Coal Mine Rescue Robots Based on Binocular Vision: A Review of the State of the Art. IEEE Access, 2020, 8: 130561-130575. [5] TSAI P, CHONDRO P, RUAN S J.Depth-Guided Pixel Dimming with Saliency-Oriented Power-Saving Transformation for Stereoscope AMOLED Displays. IEEE Transactions on Circuits and Systems for Video Technology, 2019, 30(9): 3095-3105. [6] CHONDRO P, YAO Z R, RUAN S J.Depth-Based Dynamic Lightness Adjustment Power-Saving Algorithm for AMOLED in Head-Mounted Display. Optics Express, 2018, 26(25): 33158-33165. [7] LIU P P, KING I, LYU M R, et al. Flow2Stereo: Effective Self-Supervised Learning of Optical Flow and Stereo Matching // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2020: 6647-6656. [8] HAN X F, LEUNG T, JIA Y Q, et al. MatchNet: Unifying Feature and Metric Learning for Patch-Based Matching // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2015: 3279-3286. [9] KENDALL A, MARTIROSYAN H, DASGUPTA S, et al. End-to-End Learning of Geometry and Context for Deep Stereo Regression // Proc of the IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2017: 66-75. [10] MENZE M, GEIGER A.Object Scene Flow for Autonomous Vehicles // Proc of the IEEE Conference on Computer Vision and Pa-ttern Recognition. Washington, USA: IEEE, 2015: 3061-3070. [11] SEKI A, POLLEFEYS M. Patch Based Confidence Prediction for Dense Disparity Map[C/OL]. [2021-05-26]. http://www.bmva.org/bmvc/2016/papers/paper023/paper023.pdf. [12] CHENG X L, ZHONG Y R, HARANDI M, et al. Hierarchical Neural Architecture Search for Deep Stereo Matching[C/OL].[2021-05-26]. https://arxiv.org/pdf/2010.13501v1.pdf. [13] SCHARSTEIN D, SZELISKI R.A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms. International Journal of Computer Vision, 2002, 47(1): 7-42. [14] VANNE J, AHO E, HAMALAINEN T D, et al. A High-Perfor-mance Sum of Absolute Difference Implementation for Motion Estimation. IEEE Transactions on Circuits and Systems for Video Technology, 2006, 16(7): 876-883. [15] LIU W, ANGUELOV D, ERHAN D, et al. SSD: Single Shot Mul-tibox Detector // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2016: 21-37. [16] HIRSCHMÜLLER H, INNOCENT P R, GARIBALDI J. Real-Time Correlation-Based Stereo Vision with Reduced Border Errors. International Journal of Computer Vision, 2002, 47(1): 229-246. [17] YOON K J, KWEON I S.Adaptive Support-Weight Approach for Correspondence Search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2006, 28(4): 650-656. [18] LEE J, JUN D, EEM C, et al. Improved Census Transform for Noise Robust Stereo Matching. Optical Engineering, 2016, 55(6). DOI: 10.1117/1.OE.55.6.063107. [19] MA L, LI J J, MA J, et al. A Modified Census Transform Based on the Neighborhood Information for Stereo Matching Algorithm // Proc of the 7th International Conference on Image and Graphics. New York, USA: ACM, 2013: 533-538. [20] LÜ C, LI J H, KOU Q Q, et al. Stereo Matching Algorithm Based on HSV Color Space and Improved Census Transform. Mathematical Problems in Engineering, 2021. DOI: 10.1155/2021/1857327. [21] MÜLLER M. Dynamic Time Warping // MÜLLER M, ed. Information Retrieval for Music and Motion. Berlin, Germany: Sprin-ger, 2007: 69-84. [22] HIRSCHMULLER H.Stereo Processing by Semiglobal Matching and Mutual Information. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2008, 30(2): 328-341. [23] 张大禹,黄灿.基于WTA立体匹配算法的人体检测方法研究.电子测试, 2013(5): 66-68. (ZHANG D Y, HUANG C.The Algorithm of WTA Stereo Matching Based on the Human Detection Method. Electronic Test, 2013(5): 66-68.) [24] BOBICK A F, INTILLE S S. Large Occlusion Stereo. International Journal of Computer Vision, 1999, 33(3): 181-200. [25] ZHANG K, LU J B, LAFRUIT G.Cross-Based Local Stereo Mat-ching Using Orthogonal Integral Images. IEEE Transactions on Circuits and Systems for Video Technology, 2009, 19(7): 1073-1079. [26] MEI X, SUN X, ZHOU M C, et al. On Building an Accurate Ste-reo Matching System on Graphics Hardware // Proc of the IEEE International Conference on Computer Vision Workshops. Washington, USA: IEEE, 2011: 467-474. [27] YANG Q X, WANG L, YANG R G, et al. Stereo Matching with Color-Weighted Correlation, Hierarchical Belief Propagation, and Occlusion Handling // Proc of the IEEE Computer Society Confe-rence on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2006: 2347-2354. [28] SCHARSTEIN D, SZELISKI R, ZABIH R.A Taxonomy and Eva-luation of Dense Two-Frame Stereo Correspondence Algorithms // Proc of the IEEE Workshop on Stereo and Multi-baseline Vision. Washington, USA: IEEE, 2001: 131-140. [29] SCHARSTEIN D, SZELISKI R.High-Accuracy Stereo Depth Maps Using Structured Light // Proc of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2003: 195-202. [30] YANG Q X.A Non-local Cost Aggregation Method for Stereo Mat-ching // Proc of the IEEE Conference on Computer Vision and Pa-ttern Recognition. Washington, USA: IEEE, 2012: 1402-1409. [31] FACCIOLO G, DE FRANCHIS C, MEINHARDT E. MGM: A Significantly More Global Matching for Stereovision[C/OL]. [2021-05-26]. http://www.bmva.org/bmvc/2015/papers/paper090/paper090.pdf. [32] ZHU S P, YAN L N.Local Stereo Matching Algorithm with Efficient Matching Cost and Adaptive Guided Image Filter. The Visual Computer, 2017, 33: 1087-1102. [33] VEKSLER O. Stereo Correspondence by Dynamic Programming on a Tree // Proc of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2005, II: 384-390. [34] VANETTI M, GALLO I, BINAGHI E.Dense Two-Frame Stereo Correspondence by Self-Organizing Neural Network // Proc of the 15th International Conference on Image Analysis and Processing. Berlin, Germany: Springer, 2009: 1035-1042. [35] MEI X, SUN X, DONG W M, et al. Segment-Tree Based Cost Aggregation for Stereo Matching // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2013: 313-320.
2022, Vol. 35 
