Yoichi MATSUYAMA Ph.D
Newell-Simon Hall 4604
5000 Forbes Ave, Pittsburgh, PA 15213, USA
Yoichi Matsuyama (松山 洋一) is a Post Doctoral Fellow (Special Faculty) at the ArticuLab in the School of Computer Science, Carnegie Mellon University. He has been designing and developing a number of conversational AI media systems for more than a decade. His research interest lies in computational models of human conversations, which combine artificial intelligence, social science and human-computer/robot interaction design. In the ArticuLab, he is leading SARA (Socially Aware Robot Assistant) project that was exhibited in a number of high profile conferences, such as the World Economic Forum Annual Meeting 2017, Davos, Switzerland. SARA was featured in numerous major media, such as MIT Technology Review, Washington Post, CNBC, BBC, CNET, Popular Science and Science Friday. His Ph.D dissertation project was SCHEMA, a multiparty conversation facilitation robot, specifically its computational models of facilitation strategies and language generation, as well as its robotic platform development. Prior to CMU, he was a researcher at the Perceptual Computing Group, Waseda University in Tokyo. He was also a visiting scholar in the iCub Facility (an embodied cognitive robotics research group), Italian Institute of Technology, and a committee member of ACM SIGGRAPH Asia. He received B.A. in cognitive psychology and media studies, M.E. and Ph.D in computer science from Waseda University in 2005, 2008 and 2015 respectively.
Designing Socially Expressive Conversational AI Media to Assist and Entertain Human Lives.
- Post Doctoral Research Fellow, ArticuLab, Language Technologies Institute and Human-Computer Interaction Institute, Carnegie Mellon University, Pittsburgh, United States, 2014 – present (Advisor: Justine Cassell)
- Visiting Research Fellow, iCub Facility, Italian Institute of Technology, Genova, Italy, 2013 (Advisor: Giorgio Metta)
- Director, WIZDOM (Waseda University Integrated Space of Wizards, Digital Oriented Manufacturers), 2012 – 2014
- Research Associate, Waseda University, 2010 – 2013
- Research Assistant, International Research and Education Center for Ambient Soc, Waseda University Global COE Program, 2008 – 2010
- Program Committee (Student Volunteer Program Chair), ACM SIGGRAPH Asia, 2008 – 2009
- Virtual Reality Content Designer, CAD CENTER Inc., Tokyo, 2004 – 2005
- Ph.D, Computer Science, 2015 (Advisor: Tetsunori Kobayashi, Perceptual Computing Group)
- M.E., Computer Science, 2008 (Advisor: Tetsunori Kobayashi, Perceptual Computing Group)
- B.A., Human and Social Sciences, 2005 (Advisor: Machiko Kusahara)
* All degrees from Waseda University, Tokyo, Japan
- Outstanding Research Award, Human Agent Interaction 2012, December 2012
(Yoichi Matsuyama, Akihiro Saito, Iwao Akiba, Moemi Watanabe and Tetsunori Kobayashi, Facilitation Robot Promoting the Greatest Participation of the Greatest Number in Multiparty Conversation, Human-Agent Interaction Symposium 2012, 2B-3, December 2012.)
- Best presentation, The Japanese Society for Artificial Intelligence SIG-SLUD (Special Interest Group of Speech, Language Understanding and Discourse Processing), February 2012
(Yoichi Matsuyama, Akihiro Saito, Atsushi Ito, Iwao Akiba, Moemi Watanabe and Tetsunori Kobayashi, Active Timing Detection and Strategies for Multiparty Conversation Facilitation Systems, The Japanese Society for Artificial Intelligence (JSAI), SIG-SLUD-B203-05, pp.17-24, February 2013.)
- Best presentation, The Japanese Society for Artificial Intelligence SIG-SLUD (Special Interest Group of Speech, Language Understanding and Discourse Processing), July 2008
(Yoichi Matsuyama, Shinya Fujie, Hikaru Taniyama and Tetsunori Kobayashi, Communication Activation System in Group Communication, The Japanese Society for Artificial Intelligence (JSAI), SIG-SLUD-A801, pp.15-22, July 2008.)
- Microsoft Scholarship, April 2009
- Yahoo!-CMU InMind Project, 2015-2017 … $300,000
- IT R&D program of MSIP/IITP 2017-0-00255, Autonomous Digital Companion Development, Korean Government, 2017 – 2018 … approx. $600,000
- Google Faculty Award Grant, Grounding Task Behavior in the Social World: Deep Reinforcement Learning for Social Dialogue to Improve Task Performance, 2017 … $76,109
- Google Cloud Research Credits, Socially Aware Robot Assistant, 2015 – 2017 … $50,000
- AWS Cloud Credits for Research (2018Q1): PI … $30,000
- Microsoft Grant, Socially Aware Robot Assistant, 2017 … $75,000 + Surface Hub
- CMU President Donation for SARA (2017): co-PI … $100,000
- CMU ProSEED Crosswalk Seed Grant (2018) “Holographic Archive of Research Projects (HARP) : PI … $2500
- CMU The Frank-Ratchye Fund for Art @ the Frontier, Ghost Box – Holographic Display Prototype (2018) … $500
- JSPS Grant-in-Aid for Scientific Eesearch WAKATE-B (23700239), “Development and Evaluations of Multiparty Conversation Activation Systems”, 2010 – 2012 … 3,900,000 JPY (approx. $40,000)
- JSPS Grant-in-Aid for Scientific Eesearch WAKATE-B (25870824), “Facilitation Strategy for Multiparty Conversation Robots”, 2013-2015 … 3,770,000 JPY (approx. $39,000)
- JSPS Takuetsu Graduate School Program Grant, 2014 … 9,128,641 JPY (approx. $100,000)
- Yoichi Muraoka Grant, 2012 … 1,000,000 JPY (approx. $10,000)
- Group Interaction Frontiers in Technology Workshop (GIFT), International Conference on Multimodal Interaction 2018 (ICMI 2018), Program Committee
- NAACL-HLT 2018 (North American Chapter of the Association for Computational Linguistics: Human Language Technologies), Program Committee
- ACL 2018 (Annual Meeting of the Association for Computational Linguistics), Program Committee
- IWSDS 2018 (International Workshop on Spoken Dialogue Systems Technologies), Program Committee
- Journal of Human Interface Society Japan “Human Collaboration” 2018, Associate Editor
- RO-MAN 2016 (IEEE International Symposium on Robot and Human Interactive Communication ), Associate Editor
- ACM SIGGRAPH Asia Commitee Member (2008-2009)
- SIGDIAL 2016 Special Session Organizer: The Future Directions of Dialogue-Based Intelligent Personal Assistants
- ACM SIGGRAPH Asia 2009 Special Session Organizer: THE VISION IN ROBOTICS
- IEEE Pervasive Computing, Special Issue – Conversational User Interfaces and Interactions, 2018
- Journal of Behavioral Research Methods, 2015
- International Journal of Affective Engineering, 2015
- IEEE/ACM Transactions on Acoustic, Speech and Language (TASLP), 2014
- International Workshop on Spoken Dialogue System Technologies (IWSDS) 2017, 2018
- Advanced Robotics, 2015
- IEEE-RAS International Conference on Humanoid Robots (Humanoids), 2014
- International Conference on Virtual Agents (IVA), 2014
- Journal of Japanese Society for Artificial Intelligence (JSAI), 2012, 2013
- International Conference on Social Robotics (ICSR), 2011
- Conversational Robot (Japan 2010-221556)
- Conversational Facilitation System and Robot (Japan 2008-304140)
Selected List of Research Intern Mentorship
Natural Language Understanding and Generation
- Pranav Goel (Computer Science, International Institute of Information Technology), Deep Neural Network Based Conversational Strategy Classifier, Intern 2017 Summer … now with University of Maryland.
- Wei Wang (Computer Science, Tsinghua University), Socially Conditioned Natural Language Generation using Recurrent Neural Networks: Intern 2017 Summer… now with Carnegie Mellon University.
- Vivian Tsai (Computer Science, Johns Hopkins University), Incremental Socially Conditioned Natural Language Generation: Intern 2018 Summer.
- Ying Shen (Computer Science, Fudan University), Natural Language Understanding, and Recurrent Neural Networks for Speech Markup Language (SSML) Generation: Intern 2016 Summer … now with Language Technologies Institute, Carnegie Mellon University.
- Zhao Meng (Computer Science, Peking University), Natural Language Generation using OpenCCG: Intern 2016 Summer … now with ETH Zurich.
- Akihiro Saito (Master of Computer Science, Waseda University), Automatic Expressive Opinion Sentence Generation for Enjoyable Conversational Systems: 2010 – 2012.
- Zian Zhao (Shanghai Jiao Tong University), Socially-Conditioned Task Reasoning for a Virtual Tutoring Agent: Intern 2017 Summer.
- Kweonwoo Jung (Mathematics, Carnegie Mellon University), User Simulation for Deep Reinforcement Learning Dialogue Manager of Socially Aware Conversational Agents: Independent Study 2016 Spring … now with Naver Corporation.
- Yan Ting (Electronic Information Engineering, Beihang University), Statistical Dialogue State Tracking: Intern 2016 Summer … now with Dartmouth University.
- Xuchen You (Physics, Peking University), Inverse Reinforcement Leaning for Adaptive Social Reasoning: Intern 2016 Fall.
- Iwao Akiba (Master of Computer Science, Waseda University), Facilitation Strategy Controlling Engagement Density As the Fourth Participant: 2012 – 2014 … now with Google, LLC.
- Moemi Watanabe (Department of Computer Science, Waseda University), Partially Observable Markov Decision Process (POMDP) for Engagement Management: 2012 … now with NTT Data Corporation.
- Azusa Todoroki (Department of Computer Science, Waseda University), Topic Detection and Management in Movie Recommendation Scenario: 2012 … now with Sony Corporation.
Nonverbal Behavior Understanding
- XuHai (Orson) Xu (Industrial Engineering, Tsinghua University), Automatic Listening Gesture Generation, and Computational Model of Rapport: Intern 2016 Summer.
- Divya Sai (Computational Linguistics, International Institute of Information Technology), Automatic Head-nod Detection: Intern 2016 Summer.
- Atsushi Ito (Master of Computer Science, Waseda University), Speaker and Addressee Classification using Nonverbal Cues in a Small Group: 2012 – 2014 … now with Yahoo Japan Corporation.
System Architecture Design and Integration
- Vaibhav Nachankar (Master of Information Systems Management, Carnegie Mellon University), Development of Recommendation Engine and Wizard-of-Oz User Interface, Intern 2018 Spring.
- Zihan (Atlas) Yu (Computer Engineering minor in Studio Arts, University of Pittsburgh), User Interface Infrastructure for Socially-Aware Conversational Assistant in Information Desk Scenario, Intern 2018 Spring.
- Luo Yi Tan (Entertainment Technology Center, Carnegie Mellon University), Character Development in Unity3D : Graphic Programmer 2016 … now with Unity Technologies.
- Anna Tan (Cognitive Science and Computer Science, Carnegie Mellon University), User Modeling for SARA at World Economic Forum: Independent Study 2016 Fall.
- Sara Daraei (Applied Mathematics, Sharif University of Technology), Computational Architecture for Rapport-Aligning Peer Tutoring Agent: Intern 2015 Summer … Now with University of Pittsburgh.
- Shan Han (Computer Science, Tsinghua University), System Implementation of InMind Agent User Study: Intern 2015 Summer … now with Carnegie Mellon University.
- Emily Zhou (Computer Sciece and Art, Minor in Animation and Special Effecrs), User Interface Design and Development of InMind Agent, 2015.
Human-Agent / Human-Human Data Analysis
- Deepika Mittal (Masters in Human Computer Interaction, Carnegie Mellon University), Discourse Analysis and Scenario Design of Information Desk Agent, Independent Study 2018 Spring.
- Shuangni (Annie) Huang (Humanities & Arts in Decision Science and Fine Art, Additional Major in Human-computer Interaction, Carnegie Mellon University), Use Interface Design for Socially-Aware Conversational Agent in Information Desk Scenario, Intern 2018 Spring.
- Neeti Ganjur (Science in Electrical & Computer Engineering and Engineering & Public Policy, Carnegie Mellon University), Developing Anonymization Pipeline of Human-Human Conversational Dataset, Intern 2018 Spring.
- Jiayi (Cindy) Su (Information Systems, Carnegie Mellon University), Conversational Dataset Construction for Information Desk Scenario, Intern 2018 Spring.
- Jingya Chen (Automobile Engineering, Tsinghua University), Field Study Analysis of a Socially Aware Robot Assistant at World Economic Forum,: Intern 2017 Summer … Now with Carnegie Mellon University.
- Jiajia Li (Ph.D Visiting Scholar, Digital Media Art, Communication University of China), Designing Embodied Conversatioal Intelligent Personal Assistant: 2015 – 2017 … now with Qingdao University
Demos and Exhibitions
SARA Project (Carnegie Mellon University): Project Lead
- World Economic Forum, Annual Meeting 2017 (Davos, Switzerland)
- World Economic Forum, Annual Meeting New Champions 2016 (Tianjin, China)
- White House Frontier Conference 2016
- SIGDIAL 2016 Demo Session
SCHEMA Project (Waseda University): Project Lead
SARA Project (Carnegie Mellon University): Project Lead
“Speaking with SARA certainly felt less jarring than talking to a regular chatbot. The system studies the words a person says during a conversation as well as the tone of his or her voice, also using several cameras to study the speaker’s facial expressions and head movements.”
– MIT Technology Review
- MIT Technology Review, Chatbots with Social Skills Will Convince You to Buy Something
- Google Blog (Google for Education), At Carnegie Mellon University, machine learning gets social
- BBC Business Daily, Talking to Robots
- Popular Science, S.A.R.A. Seeks To Give Artificial Intelligence People Skills
- Science Friday, Are Digital Assistants Smart Enough to Do Their Jobs?
- CNET, The Advent of Virtual Humans
- CNBC, Best of World Economic Forum in Tianjin
- Foreign Policy, “Is AI Sexist?“
- CNBC Africa, “Meet SARA, the socially-aware robot“
- Al Arabiya, “Meet Sara…the robot that knows the ins and outs of Davos“
- USA TODAY, “This robot assistant can understand facial expressions“
- CNBC, “On the ground at the Summer Davos“
- CNET, “The advent of virtual humans“
- WASHINGTON POST, The big contradiction in how the world’s most powerful people think about its future
- BLOOMBERG QUINT, SARA: A socially aware robotic assistant that reads your mood
- PressTV, Socially Aware Robot Assistant Displayed at WEF
- INDIA TIMES, Now, a socially-aware robotic assistant that gets your mood!
- THE TRIBUNE, A robotic assistant that gets your mood!
- DAILY NEWS AND ANALYSIS (DNA), Fourth industrial revolution on full display at the WEF
- FINANCIAL TIMES, Tech leaders at Davos fret over effect of AI on jobs
- JLL (Sushell Koul), From macro to machines
- CMSWIRE, Step Up Your Personalization Game with AI
- TECH FACTS LIVE, Socially Aware Robotic Assistant Introduced at World Economic Forum
- ROBOTICS AND AUTOMATION NEWS, Job-stealing robots a growing concern for world leaders
- PTS NEWS NETWORK, Socially-Aware robot frees humans from repetitive work
- DIGITAL MARKETING BUREAU, Sara: The Socially Aware Robot Assistant
- HUMAN INSIDE, S.A.R.A., the sensitive robot who improves people’s performance
- D!GITALIST MAGAZINE, Empathy: The Killer App for Artificial Intelligence
- Zee News, Here’s a socially-aware robotic assistant that gets your mood!
- The Indian Express, Now, a socially-aware robotic assistant (SARA) that gets your mood
- DE TIJD, https://twitter.com/Dorien_Luyckx/status/821731750950871040
- DW – Business, https://twitter.com/dw_business/status/821671699695435776
- CCTV (China Central Television)
- Tartan (Carnegie Mellon’s Student Newspaper) The Frontier Conference exhibits featured new technology
- Radio Sputnik, These are robots that intend to make society stronger by focusing on social bond
- Atelier.net, chatbots of growing human
- Radio Canada, Here are 4 robots at your service
- Point Park Globe, President Obama, White House host science, technology conference at CMU
SCHEMA Project (Waseda University)
- 「料理！洗濯！笑いまで！人と暮らす未来のロボット」, サイエンスZERO, 2012年4月22日 放送, NHK
- 「世界をリードするWASEDAのロボット研究」，Asahi Shimbun（朝日新聞） 2013年10月26日朝刊
- 「人とロボットの幸せな未来へ」，Yomiuri Shimbun（読売新聞） 2014年 新年号
- 「会話ロボットシェ―マ，「場を読める」目線・しぐさ」，Yomiuri Shimbun（読売新聞） 2012年 4月30日朝刊
- 「人とロボットの関係性の中に発現する知能 (SIGGRAPH Asia 2009 ロボット関連トークセッション レポート)」，Robot Watch 2009年 12月22日