Parallel Imaging: A New Theoretical Framework for Image Generation
WANG Kunfeng 1,2, LU Yue1,3, WANG Yutong1,3, XIONG Ziwei3, WANG Fei-Yue1,4
1.The State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190 2.Parallel Vision Innovation Technology Center, Qingdao Academy of Intelligent Industries, Qingdao 266000 3.School of Computer and Control Engineering, University of Chinese Academy of Sciences, Beijing 100049 4.Research Center of Military Computational Experiments and Parallel Systems, National University of Defense Technology, Changsha 410073
Abstract To build computer vision systems with good generalization ability, large-scale, diversified, and annotated image data are required for learning and evaluating the in-hand computer vision models. Since it is difficult to obtain satisfying image data from real scenes, a new theoretical framework for image generation is proposed, which is called parallel imaging. The core component of parallel imaging is various software-defined artificial imaging systems. Artificial imaging systems receive small-scale image data collected from real scenes, and then generate large amounts of artificial image data. In this paper, the realization methods of parallel imaging are summarized, including graphics rendering, image style transfer, generative models, etc. Furthermore, the characteristics of artificial images and actual images are analyzed and the domain adaptation strategies are discussed.
Fund:Supported by National Natural Science Foundation of China(No.61533019,91520301,71232006)
Corresponding Authors:
(WANG Fei-Yue(Corresponding author), born in 1961, Ph.D., professor. His research interests include modeling, analysis, and control of inte-lligent systems and complex systems.)
About author:: (WANG Kunfeng, born in 1982, Ph.D., associate professor. His research interests include intelligent transportation systems, intelligent vision computing, and machine learning.) (LU Yue, born in 1994, master student. His research interests include parallel vision, machine learning, and generative adversarial networks.) (WANG Yutong, born in 1994, Ph.D. candidate. Her research interests include computer graphics, image processing, and intelligent transportation systems.) (XIONG Ziwei, born in 1996, undergraduate student. His research interests include machine learning and image processing.) (WANG Fei-Yue(Corresponding author), born in 1961, Ph.D., professor. His research interests include modeling, analysis, and control of inte-lligent systems and complex systems.)
[1] WANG K F, LIU Y Q, GOU C, et al. A Multi-view Learning App-roach to Foreground Detection for Traffic Surveillance Applications. IEEE Transactions on Vehicular Technology, 2016, 65(6): 4144-4158. [2] WANG K F, YAO Y J. Video-Based Vehicle Detection Approach with Data-Driven Adaptive Neuro-Fuzzy Networks. International Journal of Pattern Recognition and Artificial Intelligence, 2015, 29(7). DOI: 10.1142/S0218001415550150. [3] GOU C, WANG K F, YAO Y J, et al. Vehicle License Plate Recognition Based on Extremal Regions and Restricted Boltzmann Machines. IEEE Transactions on Intelligent Transportation Systems, 2016, 17(4): 1096-1107. [4] LIU Y Q, WANG K F, SHEN D Y. Visual Tracking Based on Dynamic Coupled Conditional Random Field Model. IEEE Transactions on Intelligent Transportation Systems, 2016, 17(3): 822-833. [5] 王坤峰,苟 超,王飞跃.平行视觉:基于ACP的智能视觉计算方法.自动化学报, 2016, 42(10): 1490-1500. (WANG K F, GOU C, WANG F Y. Parallel Vision: An ACP-Based Approach to Intelligent Vision Computing. Acta Automatica Sinica, 2016, 42(10): 1490-1500.) [6] 王飞跃.平行系统方法与复杂系统的管理和控制.控制与决策, 2004, 19(5): 485-489. (WANG F Y. Parallel System Methods for Management and Control of Complex Systems. Control and Decision, 2004, 19(5): 485-489.) [7] WANG F Y. Parallel Control and Management for Intelligent Transportation Systems: Concepts, Architectures, and Applications. IEEE Transactions on Intelligent Transportation Systems, 2010, 11(3): 630-638. [8] 王飞跃.平行控制:数据驱动的计算控制方法.自动化学报, 2013, 39(4): 293-302. (WANG F Y. Parallel Control: A Method for Data-Driven and Computational Control. Acta Automatica Sinica, 2013, 39(4): 293-302.) [9] MIAO Q H, ZHU F H, L Y S, et al. A Game-Engine-Based Platform for Modeling and Computing Artificial Transportation Systems. IEEE Transactions on Intelligent Transportation Systems, 2011, 12(2): 343-353. [10] WANG F Y, ZHANG J J, ZHENG X H, et al. Where Does AlphaGo Go: From Church-Turing Thesis to AlphaGo Thesis and Beyond. IEEE/CAA Journal of Automatica Sinica, 2016, 3(2): 113-120. [11] ZHANG N, WANG F Y, ZHU F H, et al. DynaCAS: Computational Experiments and Decision Support for ITS. IEEE Intelligent Systems, 2008, 23(6): 19-23. [12] WANG F Y, WANG X, LI L X, et al. Steps toward Parallel Inte-lligence. IEEE/CAA Journal of Automatica Sinica, 2016, 3(4): 345-348. [13] TORRALBA A, EFROS A A. Unbiased Look at Dataset Bias // Proc of the IEEE Conference on Computer Vision and Pattern Re-cognition. Washington, USA: IEEE, 2011: 1521-1528. [14] MODEL I, SHAMIR L. Comparison of Data Set Bias in Object Recognition Benchmarks. IEEE Access, 2015, 3: 1953-1962. [15] 李 力,林懿伦,曹东璞,等.平行学习——机器学习的一个新型理论框架.自动化学报, 2017, 43(1): 1-8. (LI L, LIN Y L, CAO D P, et al. Parallel Learning-A New Framework for Machine Learning. Acta Automatica Sinica, 2017, 43(1): 1-8.) [16] LI L, LIN Y, ZHENG N, et al. Parallel Learning: a Perspective and a Framework. IEEE/CAA Journal of Automatica Sinica, 2017, 4(3): 389-395. [17] LUAN F J, PARIS S, SHECHTMAN E, et al. Deep Photo Style Transfer[J/OL]. [2017-06-25]. https://www.cs.cornell.edu/~fujun/files/style-cvpr17/style-cvpr17.pdf. [18] HERTZMANN A, JACOBS C E, OLIVER N, et al. Image Analogies // Proc of the 28th Annual Conference on Computer Graphics and Interactive Techniques. New York, USA: ACM, 2001: 327-340. [19] OKURA F, VANHOEY K, BOUSSEAU A, et al. Unifying Color and Texture Transfer for Predictive Appearance Manipulation // Proc of the Eurographics Symposium on Rendering. Berlin, Germany: Springer, 2015: 53-63. [20] EFROS A A, FREEMAN W T. Image Quilting for Texture Synthesis and Transfer // Proc of the 28th Annual Conference on Compu-ter Graphics and Interactive Techniques. New York, USA: ACM, 2001: 341-346. [21] ASHIKHMIN N. Fast Texture Transfer. IEEE Computer Graphics and Applications, 2003, 23(4): 38-43. [22] GATYS L A, ECKER A S, BETHGE M. Image Style Transfer Using Convolutional Neural Networks // Proc of the IEEE Confe-rence on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 2414-2423. [23] 王坤峰,苟 超,段艳杰,等.生成式对抗网络GAN的研究进展与展望.自动化学报, 2017, 43(3): 321-332. (WANG K F, GOU C, DUAN Y J, et al. Generative Adversarial Networks: The State of the Art and Beyond. Acta Automatica Sinica, 2017, 43(3): 321-332.) [24] GOODFELLOW I J, POUGET-ABADIE J, MIRZA M, et al. Ge-nerative Adversarial Nets // GHAHRAMANI Z, WELLING M, CORTES C, et al., eds. Advances in Neural Information Processing Systems 27. Cambridge, USA: The MIT Press, 2014: 2672-2680. [25] BERTHELOT D, SCHUMM T, METZ L. BEGAN: Boundary Equilibrium Generative Adversarial Networks[C/OL]. [2017-06-25]. https://arxiv.org/pdf/1703.10717.pdf [26] ZHAO J B, MATHIEU M, LECUN Y. Energy-Based Generative Adversarial Network[C/OL]. [2017-06-25]. https://arxiv.org/pdf/1609.03126.pdf. [27] ARJOVSKY M, CHINTALA S, BOTTOU L. Wasserstein GAN [C/OL]. [2017-06-25]. https://arxiv.org/pdf/1701.07875v1.pdf. [28] SHRIVASTAVA A, PFISTER T, TUZEL O, et al. Learning from Simulated and Unsupervised Images through Adversarial Training[C/OL]. [2017-06-25]. https://arxiv.org/pdf/1612.07828.pdf. [29] ZHU J Y, PARK T, ISOLA P, et al. Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks[C/OL]. [2017-06-25]. https://arxiv.org/pdf/1703.10593.pdf [30] CHEN W Z, WANG H, LI Y Y, et al. Synthesizing Training Images for Boosting Human 3D Pose Estimation // Proc of the 4th International Conference on 3D Vision. Washington, USA: IEEE, 2016: 479-488. [31] GANIN Y, USTINOVA E, AJAKAN H, et al. Domain-Adversarial Training of Neural Networks. Journal of Machine Learning Research, 2016, 17: 1-35. [32] GHIFARY M. Domain Adaptation and Domain Generalization with Representation Learning. Ph.D Dissertation. Wellington, New Zealand: Victoria University of Wellington, 2016. [33] ROS G, SELLART L, MATERZYNSKA J, et al. The SYNTHIA Dataset: A Large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 3234-3243. [34] MOVSHOVITZ-ATTIAS Y, KANADE T, SHEIKH Y. How Useful Is Photo-Realistic Rendering for Visual Learning? [C/OL]. [2017-06-25]. https://arxiv.org/pdf/1603.08152.pdf. [35] V ZQUEZ D, L PEZ A M, MAR N J, et al. Virtual and Real World Adaptation for Pedestrian Detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014, 36(4): 797-809. [36] XU J L, V ZQUEZ D, L PEZ AM, et al. Learning a Part-Based Pedestrian Detector in a Virtual World. IEEE Transactions on Intelligent Transportation Systems, 2014, 15(5): 2121-2131. [37] XU J L, RAMOS S, V ZQUEZ D, et al. Domain Adaptation of Deformable Part-Based Models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014, 36(12): 2367-2380. [38] HANDA A, P TR UCEAN V, BADRINARAYANAN V, et al. Understanding Real World Indoor Scenes with Synthetic Data[C/OL]. [2017-06-25]. https://arxiv.org/pdf/1511.07041.pdf. [39] GAIDON A, WANG Q, CABON Y, et al. Virtual Worlds as Proxy for Multi-object Tracking Analysis // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 4340-4349. [40] 王飞跃,莫 红.关于二型模糊集合的一些基本问题.自动化学报, 2017, 43(7): 1114-1141. (WANG F Y, MO H. Some Fundamental Issues on Type-2 Fuzzy Sets. Acta Automatica Sinica, 2017, 43(7): 1114-1141.)
2017, Vol. 30 
