Research and Development of Cognitive Computing in Mind
WANG Zhi-Liang1 , ZHENG Si-Yi1,2, WANG Xian-Mei1 , WANG Wei1
1.School of Information Engineering, University of Science and Technology Beijing, Beijing 100083 2.Beijing Command College of Chinese Peoples Armed Police Force, Beijing 100012
Abstract Cognitive computing in mind is an important part of intelligent human-computer interaction (HCI), which attracts increasing attention from researchers in recent years. The development of the research advances at home and abroad is surveyed. Firstly, the conceptions correlated to cognitive computing in mind are introduced, and then the mechanisms and research contents in mind-reading are described in detail. Secondly, main neurobiological achievements of mental cognition are summarized and the research status of cognitive state in mind is compared to affective state. Moreover, the application trends of mental cognition in HCI are analyzed from the aspects of model establishment and mode extraction. Then, the primary frame of vision cognitive computing model of mental state with multi-level and multi-mode information fusion is put forward. The existing problems and the significance of cognitive computing in mind are finally discussed.
[1] Premack D, Woodruff G. Does the Chimpanzee Have a Theory of Mind? The Behavior and Brain Sciences, 1978, 4(4): 515-526 [2] Turk M. RTV4HCI:A Historical Overview // Kisacˇanin B, Pavlovic＇ V, Huang T, eds. Real-time Vision for Human-Computer Interaction. New York, USA: Springer-Verlag, 2005: 3-13 [3] Guo Liyan. Psychology. Nanjing, China: Nanjing University Press, 2006 (in Chinese) (郭黎岩.心理学.南京:南京大学出版社, 2006) [4] Baron-Cohen S, Pippa C. Reading the Eyes: Evidence for the Role of Perception in the Development of a Theory of Mind. Mind and Language, 1992, 7(1/2): 172-186 [5] Baron-Cohen S. Theory of Mind and Autism: A Review. International Review of Research in Mental Retardation, 2000, 23: 169-184 [6] Realo A, Allik J. Mind-Reading Ability: Beliefs and Performance. Journal of Research in Personality, 2003, 37(5): 420-445 [7] Elliot A, Timothy D, Robin M. Social Psychology. 5th Edition. Upper Saddle River, USA: Prentice Hall, 2004 [8] Baron-Cohen S, Riviere A, Fukushima M, et al. Reading the Mind in the Face: A Cross-Cultural and Developmental Study. Visual Cognition, 1996, 3(1): 39-60 [9] Howlin P, Baron-Cohen S, Hadwin J. Teaching Children with Autism to Mind-Read: A Practical Guide for Teachers and Parents. New York, USA: John Wiley and Sons, 1999 [10] Wang Geng, Wang Ansheng. Cognitive Psychology. Beijing, China: Peking University Press, 2001 (in Chinese) (王 埂,汪安圣.认知心理学.北京:北京大学出版社, 2001) [11] Tager-Flusberg H, Sullivan K. A Componential View of Theory of Mind: Evidence from Williams Syndrome. Cognition, 2000, 76(1): 59-90 [12] Sabbagh M A. Understanding Orbitofrontal Contributions to Theory-of-Mind Reasoning: Implications for Autism. Brain and Cognition, 2004, 55(1): 209-219 [13] Brian C S. The Foundations of Computing // Scheutz M, ed. Computationalism. Cambridge, USA: MIT Press, 2002: 23-58 [14] Smith S. What is Cognitive Computing [EB/OL]. [2010-05-25]. http://www.wisegeek.com/what-is-cognitive- computing.htm [15] Berkeley U C. Cognitive Computing 2007 [EB/OL]. [2010-05-25]. http://www-bisc.eecs.berkeley.edu/CognitiveComputing07/cognitivecomputing07.pdf [16] Valiant L. Cognitive Computation // Proc of the 36th Annual Symposium on Foundations of Computer Science. Milwaukee, USA, 1995: 2-3 [17] Luria A. The Working Brain: An Introduction to Neuropsychology. New York, USA: Basic Books, 1976 [18] Zeki S. A Vision of the Brain. Oxford, UK: Blackwell Scientific Publications, 1993 [19] Gazzaniga M. The Cognitive Neurosciences. Cambridge, USA: MIT Press, 2004 [20] Gallagher H, Happe F, Fletcher P. Reading the Mind in Cartoons and Stories: An fMRI Study of ‘Theory of Mind’ in Verbal and Nonverbal Tasks. Neuropsychologia, 2000, 38(1): 11-21 [21] Wei Jinghan, Luo Yuejia. The Cognitive ERP Textbook. Beijing, China: The Economic Daily Press, 2002 (in Chinese) (魏景汉,罗跃嘉.认知事件相关脑电位教程.北京:经济日报出版社, 2002) [22] Woldorff M G, Gallen C C, Hampson S A, et al. Modulation of Early Sensory Processing in Human Auditory Cortex during Auditory Selective Attention. Proc of the National Academy of Sciences of the USA, 1993, 90(18): 8722-8726 [23] Raichle M E. Visualizing the Mind. Scientific American, 1994, 270(4): 36-43 [24] Wheeler M, Stuss D, Tulving E. Toward a Theory of Episodic Memory: The Frontal Lobes and Autonoetic Consciousness. Psychological Bulletin, 1997, 121(3): 331-354 [25] Anderson S J, Holliday I E, Singh K D, et al. Localization and Functional Analysis of Human Cortical Area V5 Using Magneto-Encephalography. Proc of the Royal Society, 1996, 263(1369): 423-431 [26] LeDoux J. The Emotional Brain: the Mysterious Underpinnings of Emotional Life. New York, USA: Simon Schuster, 1996 [27]Ralph A. Recognizing Emotion from Facial Expressions: Psychological and Neurological Mechanisms. Behavioral and Cognitive Neuroscience Reviews, 2002, 1(1): 21-62 [28] Purves D, Augustine G J. Emotions // Purves D, Augustine G J, Fitzpatrick D, eds. Neuroscience. 2nd Edition. Sunderland, USA: Sinauer Associates, 2001: 2030-2076 [29] Norbert S. Emotion, Cognition and Decision Making. Cognition and Emotion, 2000, 14(4): 433-440 [30] Picard R. Affective Computing. Cambridge, USA: MIT Press, 1997 [31] Wilson G M, Sasse M A. From Doing to Being: Getting Closer to the User Experience. Interacting with Computers, 2004, 16(4): 697-705 [32] Picard R W, Scheirer J. The Galvactivator: A Glove that Senses and Communicates Skin Conductivity // Proc of the International Conference on Human-Computer Interaction. New Orleans, USA, 2001: 91-101 [33] Scheirer J, Fernandez R, Klein J, et al. Frustrating the User on Purpose: A Step toward Building an Affective Computer. Interacting with Computers, 2002, 14(2): 93-118 [34] Qi Yuan, Picard R W. Context-Sensitive Bayesian Classifiers and Application to Mouse Pressure Pattern Classification // Proc of the 16th International Conference on Pattern Recognition. Quebec, Canada, 2002, VIII: 448-451 [35] Sebe N, Lew M S, Sun Y, et al. Authentic Facial Expression Analysis. Image and Vision Computing, 2007, 25(12): 1856-1863 [36] Picard R, Papert S, Bender W, et al. Affective Learning-A Manifesto. BT Technology Journal, 2005, 22(4): 253-269 [37] Kort B, Reilly R, Picard R. An Affective Model of Interplay between Emotions and Learning: Reengineering Educational Pedagogy Building a Learning Companion // Proc of the IEEE International Conference on Advanced Learning Technologies. Madison, USA, 2001: 43-46 [38] Klein J, Moon Y, Picard R W. This Computer Responds to User Frustration: Theory, Design, and Results. Interacting with Computers, 2000, 14(2): 119-140 [39] Conati C. Probabilistic Assessment of Users Emotions in Educational Games. Journal of Applied Artificial Intelligence, 2002, 16(7/8): 555-575 [40] Conati C, Gertner A, VanLehn K. Using Bayesian Networks to Manage Uncertainty in Student Modeling. Journal of User Modeling and User-Adapted Interaction, 2002, 12(4): 371-417 [41] Lisetti C, Nasoza F, LeRouge C, et al. Developing Multimodal Intelligent Affective Interfaces for Tele-home Health Care. International Journal of Human-Computer Studies, 2003, 59(1/2): 245-255 [42] Littlewort G, Bartlett M, Fasel I, et al. Towards Social Robots: Automatic Evaluation of Human-Robot Interaction by Face Detection and Expression Classification // Thrun S, Saul L K, Schlkopf B, eds. Advances in Neural Information Processing Systems. Cambridge, USA: MIT Press, 2004, XVI: 1563-1570 [43] Breazeal C. Emotion and Sociable Humanoid Robots. International Journal of Human-Computer Studies, 2003, 59(1/2): 119-155 [44] Fong T, Nourbakhsh I, Dautenhahn K. A Survey of Socially Interactive Robots, Robotics and Autonomous Systems. Journal of Nonverbal Behavior, 2003, 42(3/4): 143-166 [45] Dautenhahn K, Billard A. Games Children with Autism Can Play with Robota, a Humanoid Robotic Doll // Proc of the 1st Cambridge Workshop on Universal Access and Assistive Technology. London, UK, 2002: 179-190 [46] Paiva A, Costaa M, Chavesa R. SenToy: An Affective Sympathetic Interface. International Journal of Human-Computer Studies, 2003, 59(1/2): 227-235 [47]Garau M, Slater M, Vinayagamoorthy V, et al. The Impact of Avatar Realism and Eye Gaze Control on Perceived Quality of Communication in a Shared Immersive Virtual Environment // Proc of the SIGCHI Conference on Human Factors in Computing Systems. Fort Lauderdale, USA, 2003: 529-536 [48] Xue Yuli, Mao Xia, Guo Ye, et al. The Research Advance of Facial Expression Recognition in Human Computer Interaction. Journal of Image and Graphics, 2009, 14(5): 764-772 (in Chinese) (薛雨丽,毛 峡,郭 叶,等.人机交互中的人脸表情识别研究进展. 中国图象图形学报, 2009, 14(5): 764-772) [49] Zhang Baoguo, Song Qinghua, Fei Shumin. Study of Emotional Speech Recognition. Computer Technology and Development, 2009, 19(1): 92-96 (in Chinese) (章国宝,宋清华,费树岷.语音情感识别研究.计算机技术与发展, 2009, 19(1): 92-96) [50] Huang Lixing, Xin Le, Zhao Liyue, et al. Bimodal Emotion Recognition Based on Adaptive Weights. Journal of Tsinghua University: Science and Technology, 2008, 48(Z1): 715-719 (in Chinese) (黄力行,辛 乐,赵礼悦,等.自适应权重的双模态情感识别.清华大学学报:自然科学版, 2008, 48(Z1): 715-719) [51] Jin Hui, Gao Wen. The Human Facial Combined Expression Recognition System. Chinese Journal of Computers, 2000, 23(6): 602-608 (in Chinese) (金 辉,高 文.人脸面部混合表情识别系统.计算机学报, 2000, 23(6): 602-608) [52] Mao Xia, Xue Yuli, Zhang Fan. Design and Realization of BHU Facial Expression Database. Journal of Beijing University of Aeronautics and Astronautics, 2007, 33(2): 224-228 (in Chinese) (毛 峡,薛雨丽,张 帆.BHU人脸表情数据库的设计与实现.北京航空航天大学学报, 2007, 33(2): 224-228) [53] Zhu Yongchong. Research on Emotional Feature Analysis and Multi Sub-Pattern Voting in Speech Signal Recognition. Master Dissertation. Harbin, China: Harbin Institute of Technology. School of Computer Science and Technology, 2005 (in Chinese) (朱永崇.语音情感识别的特征分析与多子模式投票方法的研究.硕士学位论文.哈尔滨:哈尔滨工业大学.计算机科学与技术学院, 2005) [54] Wei Ran, Jiang Li, Tao Linmi. Facial Expression Recognition System Based on Multiple Feature Integration. Journal of Image and Graphics, 2009, 14(5): 792-800 (in Chinese) (魏 冉,姜 莉,陶霖密.融合人脸多特征信息的表情识别系统.中国图象图形学报, 2009, 14(5): 792-800) [55] Xu Shuang, Jia Yunde. Facial Expression Manifold Based on Expression Similarity. Journal of Software, 2009, 20(8): 2191-2198 (in Chinese) (续 爽,贾云得.基于表情相似性的人脸表情流形.软件学报, 2009, 20(8): 2191-2198) [56]Wang Wei, Wang Zhiliang, Zheng Siyi, et al. Affective Model in Human-Robot Interaction. CAAI Trans on Intelligent Systems, 2010, 5(1): 10-16 (in Chinese) (王 巍,王志良,郑思仪,等.人机交互中的个性化情感模型. 智能系统学报, 2010, 5(1): 10-16) [57] Wang Zhiliang. Artificial Psychology-A most Accessible Science Research to Human Brain. Journal of University of Science and Technology Beijing, 2000, 22(5): 478-481 (in Chinese) (王志良.人工心理学——关于更接近人脑工作模式的科学.北京科技大学学报, 2000, 22(5): 478-481) [58] Zeng Z, Pantic M, Roisman G I, et al. A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions. IEEE Trans on Pattern Analysis and Machine Intelligence, 2009, 31(1): 39-58 [59] La S K, Ashley C, Boord P, et al. Development of An Algorithm for an EEG-Based Driver Fatigue Counter Measure. Journal of Safety Research, 2003, 34(3): 321-328 [60] Katsis C, Ntouvas N, Bafas C, et al. Assessment of Muscle Fatigue during Driving Using Surface EMG // Proc of the IASTED International Conference on Biomedical Engineering. Innsbruck, Austria, 2004: 259-262 [61] Lin C T, Lin Hongzhang, Chiu T W, et al. Distraction-Related EEG Dynamics in Virtual Reality Driving Simulation // Proc of the IEEE International Symposium on Circuits and Systems. Seattle, USA, 2008: 1088-1091 [62] Greeley H P, Friets E, Wilson J P, et al. Detecting Fatigue from Voice Using Speech Recognition // Proc of the IEEE International Symposium on Signal Processing and Information Technology. Vancouver, Canada, 2006: 567-571 [63] Neiberg D, Elenius K, Laskowski K. Emotion Recognition in Spontaneous Speech Using GMM // Proc of the 9th International Conference on Spoken Language. Pittsburgh, USA, 2006: 809-812 [64] Saradadevi M, Bajaj P. Driver Fatigue Detection Using Mouth and Yawning Analysis. International Journal of Computer Science and Network Security, 2008, 8(6): 183-188 [65] Liang Yulan, Reyes M L, Lee J D. Real-Time Detection of Driver Cognitive Distraction Using Support Vector Machines. IEEE Trans on Intelligent Transportation Systems, 2007, 8(2): 340-350 [66] Gan Ling, Cui Bing, Wang Weixing. Driver Fatigue Detection Based on Eye Tracking // Proc of the 6th World Congress on Intelligent Control and Automation. Dalian, China, 2008: 5341-5344 [67] Yu Chunqing. Drivers Posture Monitoring Analysis Based on Two Asymmetrical Laying Aside Cameras. Master Dissertation. Nanjing, China: Nanjing University of Science and Technology. School of Computer Science and Technology, 2007 (in Chinese) (于春青.基于非对称放置双目摄像头的驾驶员姿态监控分析.硕士学位论文.南京:南京理工大学.计算机科学与技术学院, 2007) [68] Chen L S H. Joint Processing of Audio-Visual Information for the Recognition of Emotion Expressions in Human-Computer Interaction. Ph.D Dissertation. Urbana, USA: University of Illinois at Urbana-Champaign. The Graduate College, 2000 [69] Baron-Cohen S. How to Build a Baby that can Read Minds: Cognitive Mechanisms in Mindreading. Current Psychology of Cognition, 1994, 13(5): 513-552 [70] Kutila M, Jokela M, Markkula G, et al. Driver Distraction Detection with a Camera Vision System // Proc of the IEEE International Conference on Image Processing. San Antonio, USA, 2007, VI: 201-204 [71] Damousis I G, Tzovaras D. Fuzzy Fusion of Eyelid Activity Indicators for Hypovigilance-Related Accident Prediction. IEEE Trans on Intelligent Transportation System, 2008, 9(3): 491-500 [72] Ji Qiang, Zhu Zhiwei, Lan P. Real-Time Nonintrusive Monitoring and Prediction of Driver Fatigue. IEEE Trans on Vehicular Technology, 2004, 53(4): 1052-1068 [73] A Mao. Robot Tend Towards to Cognition. Chinese Newspaper Week, 2008, (48): 58-59 (in Chinese) (阿 茂.走向认知的机器人.中国新闻周刊, 2008, (48): 58-59) [74] Kaliouby R, Robinson P. Mind Reading Machines: Automated Inference of Cognitive Mental States from Video // Proc of the International Conference on System, Man and Cybernetics. The Hague, Netherlands, 2004, I: 682-688 [75]Sobol-Shikler T, Robinson P. Classification of Complex Information: Inference of Co-Occurring Affective States from Their Expressions in Speech. IEEE Trans of Pattern Analysis and Machine Intelligence, 2010, 32(7): 1284-1297 [76] Haidt J, Keltner D. Culture and Facial Expression: Open-Ended Methods Find More Expressions and a Gradient of Recognition. Cognition and Emotion. 1999, 13(3): 225-266 [77] Langton, S R H, Watt R J, Bruce V. Do the Eyes Have It? Cues to the Direction of Social Attention. Trends in Cognitive Sciences, 2000, 4(2): 50-59 [78] Baron-Cohen S, Golan O, Wheelwright S. Mind Reading: The Interactive Guide to Emotions. London, UK: Jessica Kingsley Publishers, 2004 [79] Ungerleider L, Mishkin M. Two Cortical Visual System // Ingle D, Goodale M, Mansfield R, eds. Analysis of Visual Behavior. Cambridge, USA: MIT Press, 1982: 549-586 [80] Gao Jun, Xie Zhao. Image Understanding Theory and Approach. Beijing, China: Science Press, 2009 (in Chinese) (高 隽,谢 昭.图像理解理论与方法.北京:科学出版社, 2009) [81] Fletcher P C. Happ F, Frith U, et al. Other Minds in the Brain: A Functional Imaging Study of Theory of Mind in Story Comprehension. Cognitive, 1995, 57(2): 109-128 [82] Cowie R, Douglas-Cowie E, Tsapatsoulis N. Emotion Recognition in Human-Computer Interaction. IEEE Signal Processing Magazine, 2001, 18(1): 32-80
2011, Vol. 24 
