Abstract The traditional video stabilization algorithms cannot achieve good performance with a low latency. Aiming at this problem, an algorithm is proposed using the Lie group manifold based Kalman filter to stabilize videos in real time. The video frame motion is separated into rotation component and translation component. The rotation component can be represented by the rotation matrix obtained via the gyroscope data, while the translation component is provided by the translation matrix computed by the matching between video frames. Both the sequence of the rotation matrices and the translation matrices can form the motion paths on the Lie group manifold. Therefore, the Lie group manifold based Kalman filter is used to smooth the rotation and translation component, respectively. Finally, the video frame sequence can be stabilized by the motion compensation. The experimental results show that the proposed algorithm achieves high online real-time video stabilization performance on the mobile device.
Fund:Supported by National Natural Science Foundation of China(No.61772069)
About author:: YANG Jiali, master student, Her research interests include image and video proce-ssing.LAI Linjing(Corresponding author), master, assistant researcher. Her research inte-rests include image and video processing and virtual reality.ZHANG Lei, Ph.D., associate professor. His research interests include image and video processing.HUANG Hua, Ph.D., professor. His resear-ch interests include image and video proce-ssing.
[1] PUGLISI G, BATTIATO S. A Robust Image Alignment Algorithm for Video Stabilization Purposes. IEEE Transactions on Circuits and Systems for Video Technology, 2011, 21(10): 1390-1400. [2] 陈启立,宋 利,余松煜.视频稳像技术综述.电视技术, 2011, 35(7): 15-17. (CHEN Q L, SONG L, YU S Y. Overview of Video Stabilization Technology. Television Technology, 2011, 35(7): 15-17.) [3] MATSUSHITA Y, OFEK E, GE W N, et al. Full-Frame Video Stabilization with Motion Inpainting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2006, 28(7): 1150-1163. [4] CHEN B Y, LEE K Y, HUANG W T, et al. Capturing Intention-Based Full-Frame Video Stabilization. Computer Graphics Forum, 2008, 27(7): 1805-1814. [5] LEE K Y, CHUANG Y Y, CHEN B Y, et al. Video Stabilization using Robust Feature Trajectories // Proc of the 12th IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2009: 1397-1404. [6] GOLDSTEIN A, FATTAL R. Video Stabilization Using Epipolar Geo-metry. ACM Transactions on Graphics, 2012, 31(5), DOI: 10.1145/2231816.2231824. [7] WANG Y S, LIU F, HSU P S, et al. Spatially and Temporally Optimized Video Stabilization. IEEE Transactions on Visualization and Computer Graphics, 2013, 19(8): 1354-1361. [8] GRUNDMANN M, KWATRA V, ESSA I. Auto-Directed Video Stabilization with Robust L1 Optimal Camera Paths // Proc of the IEEE Interntational Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2011: 225-232. [9] LIU F, GLEICHER M, WANG J, et al. Subspace Video Stabilization. ACM Transactions on Graphics, 2011, 30(1). DOI: 10.1145/1899404.1899408. [10] LIU S C, YUAN L, TAN P, et al. Bundled Camera Paths for Vi-deo Stabilization. ACM Transactions on Graphics, 2013, 32(4). DOI: 10.1145/2461912.2461995. [11] CENSI A, FUSIELLO A, ROBERTO V. Image Stabilization by Feature Stracking // Proc of the 10th International Conference on Image Analysis and Processing. Washington, USA: IEEE, 1999: 665-667. [12] HU R, SHI R J SHEN I F, et al. Video Stabilization Using Scale-Invariant Features // Proc of the 11th International Conference on Information Visualization. Washington, USA: IEEE, 2007: 871-877. [13] ZHOU M Q, ASARI V K. A Fast Video Stabilization System Based on Speeded-up Robust Features // Proc of the International Symposium on Visual Computing. Berlin, Germany: Springer, 2011: 428-435. [14] KARPENKO A, JACOBS D, BAEK J, et al. Digital Video Stabilization and Rolling Shutter Correction Using Gyroscopes[T/OL]. [2018-12-15]. http://movement.stanford.edu/papers/stabilization/karpenko_gyro.pdf. [15] HANNING G, FORSLÖW N, FORSSÉN P E, et al. Stabilizing Cell Phone Video Using Inertial Measurement Sensors // Proc of the IEEE International Conference on Computer Vision Workshops. Washington, USA: IEEE, 2011: 1-8. [16] 李凡长,康 宇.基于Lie群的机器学习理论框架.云南民族大学学报(自然科学版), 2004, 13(4): 251-255. (LI F Z, KANG Y. The Study of Machine Learning Theory Frame Based on Lie Group. Journal of Yunnan Nationalities University(Natural Sciences Edition), 2004, 13(4): 251-255.) [17] 李凡长,钱旭培,谢 琳,等.机器学习理论及应用.北京:中国科学技术大学出版社, 2009. (LI F Z, QIIAN X P, XIE L, et al. Machine Learning and Its App-lications. Beijing, China: University of Science and Technology of China Press, 2009.) [18] ROS G, GUERRERO J, SAPPA A D, et al. VSLAM Pose Initia-lization via Lie Groups and Lie Algebras Optimization // Proc of the IEEE International Conference on Robotics and Automation. Washington, USA: IEEE, 2013: 5720-5727. [19] JIA C, EVANS B L. Constrained 3D Rotation Smoothing via Glo-bal Manifold Regression for Video Stabilization. IEEE Transactions on Signal Processing, 2014, 62(13): 3293-3304. [20] ZHANG L, CHEN X Q, KONG X Y, et al. Geodesic Video Stabilizationin Transformation Space. IEEE Transactions on Image Processing, 2017, 26(5): 2219-2229. [21] HELGASON S. Differential Geometry, Liegroups, and Symmetric Spaces. Providence, USA: American Mathematical Society, 2001. [22] WELCH G, BISHOP G. An Introduction to the Kalman Filter[C/OL]. [2018-12-18]. https://courses.cs.washington.edu/courses/cse571/00au/papers/welch-bishop-kalman.pdf. [23] LI L Y, MA X H, ZHAO Z N. Real-Time Video Stabilization Based on Fast Block Matching and Improved Kalman Filter // Proc of the 5th International Conference on Intelligent Control and Information Processing. Washington, USA: IEEE, 2014: 394-397. [24] ERTURK S. Real-Time Digital Image Stabilization Using Kalman Flters. Real-Time Imaging, 2002, 8(4): 317-328. [25] LIU S C, TAN P, YUAN L, et al. Meshflow: Minimum Latency Online Video Stabilization // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2016: 800-815. [26] ZHANG L, ZHENG Q Z, HUANG H. Intrinsic Motion Stability Assessment for Video Stabilization. IEEE Transactions on Visua-lization and Computer Graphics, 2019, 25(4): 1681-1692.
2019, Vol. 32 
