Yoichi MATSUYAMA Ph.D

School of Computer Science, Carnegie Mellon University
Newell-Simon Hall 4604
5000 Forbes Ave, Pittsburgh, PA 15213, USA

CV (June 2018)
Conversational AI Media Design Portfolio

 

Yoichi Matsuyama (松山 洋一) is a Post Doctoral Fellow (Special Faculty) at the ArticuLab in the School of Computer Science, Carnegie Mellon University. He has been designing and developing a number of conversational AI media systems for more than a decade. His research interest lies in computational models of human conversations, which combine artificial intelligence, social science and human-computer/robot interaction design. In the ArticuLab, he is leading SARA (Socially Aware Robot Assistant) project that was exhibited in a number of high profile conferences, such as the World Economic Forum Annual Meeting 2017, Davos, Switzerland. His Ph.D dissertation project was SCHEMA, a multiparty conversation facilitation robot, specifically its computational models of facilitation strategies and language generation, as well as its robotic platform development. Prior to CMU, he was a researcher at the Perceptual Computing Group, Waseda University in Tokyo. He was also a visiting researcher in the iCub Facility, Italian Institute of Technology, and a committee member of ACM SIGGRAPH Asia. He received B.A. in cognitive psychology and media studies, M.E. and Ph.D in computer science from Waseda University in 2005, 2008 and 2015 respectively.

 

Mission

Designing Socially Expressive Conversational AI Media to Assist and Entertain Human Lives.

 

 

Experiences

  • Post Doctoral Research Fellow, ArticuLab, Language Technologies Institute and Human-Computer Interaction Institute, Carnegie Mellon University, Pittsburgh, United States, 2014 – present (Advisor: Justine Cassell)
  • Visiting Research Fellow, iCub Facility, Italian Institute of Technology, Genova, Italy, 2013 (Advisor: Giorgio Metta)
  • Director, WIZDOM (Waseda University Integrated Space of Wizards, Digital Oriented Manufacturers), 2012 – 2014
  • Research Associate, Waseda University, 2010 – 2013
  • Research Assistant, International Research and Education Center for Ambient Soc, Waseda University Global COE Program, 2008 – 2010
  • Program Committee (Student Volunteer Program Chair), ACM SIGGRAPH Asia, 2008 – 2009
  • Virtual Reality Content Designer, CAD CENTER Inc., Tokyo, 2004 – 2005

Educations

  • Ph.D, Computer Science, 2015 (Advisor: Tetsunori Kobayashi, Perceptual Computing Group)
  • M.E., Computer Science, 2008 (Advisor: Tetsunori Kobayashi, Perceptual Computing Group)
  • B.A., Human and Social Sciences, 2005 (Advisor: Machiko Kusahara)

* All degrees from Waseda University, Tokyo, Japan

Awards

  • Outstanding Research Award, Human Agent Interaction 2012, December 2012
    (Yoichi Matsuyama, Akihiro Saito, Iwao Akiba, Moemi Watanabe and Tetsunori Kobayashi, Facilitation Robot Promoting the Greatest Participation of the Greatest Number in Multiparty Conversation, Human-Agent Interaction Symposium 2012, 2B-3, December 2012.)
  • Best presentation, The Japanese Society for Artificial Intelligence SIG-SLUD (Special Interest Group of Speech, Language Understanding and Discourse Processing), February 2012
    (Yoichi Matsuyama, Akihiro Saito, Atsushi Ito, Iwao Akiba, Moemi Watanabe and Tetsunori Kobayashi, Active Timing Detection and Strategies for Multiparty Conversation Facilitation Systems, The Japanese Society for Artificial Intelligence (JSAI), SIG-SLUD-B203-05, pp.17-24, February 2013.)
  • Best presentation, The Japanese Society for Artificial Intelligence SIG-SLUD (Special Interest Group of Speech, Language Understanding and Discourse Processing), July 2008
    (Yoichi Matsuyama, Shinya Fujie, Hikaru Taniyama and Tetsunori Kobayashi, Communication Activation System in Group Communication, The Japanese Society for Artificial Intelligence (JSAI), SIG-SLUD-A801, pp.15-22, July 2008.)
  • Microsoft Scholarship, April 2009

Grants

  • Yahoo!-CMU InMind Project, 2015-2017 … $300,000
  • IT R&D program of MSIP/IITP 2017-0-00255, Autonomous Digital Companion Development, Korean Government, 2017 – 2018 … approx. $600,000
  • Google Faculty Award Grant, Grounding Task Behavior in the Social World: Deep Reinforcement Learning for Social Dialogue to Improve Task Performance, 2017 … $76,109
  • Google Cloud Research Credits, Socially Aware Robot Assistant, 2015 – 2017 … $50,000
  • AWS Cloud Credits for Research (2018Q1): PI … $30,000
  • Microsoft Grant, Socially Aware Robot Assistant, 2017 … $75,000 + Surface Hub
  • CMU President Donation for SARA (2017): co-PI … $100,000
  • CMU ProSEED Crosswalk Seed Grant (2018) “Holographic Archive of Research Projects (HARP) : PI … $2500
  • CMU The Frank-Ratchye Fund for Art @ the Frontier, Ghost Box – Holographic Display Prototype (2018) … $500
  • JSPS Grant-in-Aid for Scientific Eesearch WAKATE-B (23700239), “Development and Evaluations of Multiparty Conversation Activation Systems”, 2010 – 2012 … 3,900,000 JPY (approx. $40,000)
  • JSPS Grant-in-Aid for Scientific Eesearch WAKATE-B (25870824), “Facilitation Strategy for Multiparty Conversation Robots”, 2013-2015 … 3,770,000 JPY (approx. $39,000)
  • JSPS Takuetsu Graduate School Program Grant, 2014 … 9,128,641 JPY (approx. $100,000)
  • Yoichi Muraoka Grant, 2012 … 1,000,000 JPY (approx. $10,000)

Professional Services

Program Committee

  • Group Interaction Frontiers in Technology Workshop (GIFT), International Conference on Multimodal Interaction 2018 (ICMI 2018), Program Committee
  • NAACL-HLT 2018 (North American Chapter of the Association for Computational Linguistics: Human Language Technologies), Program Committee
  • ACL 2018 (Annual Meeting of the Association for Computational Linguistics), Program Committee
  • IWSDS 2018 (International Workshop on Spoken Dialogue Systems Technologies), Program Committee
  • Journal of Human Interface Society Japan “Human Collaboration” 2018, Associate Editor
  • RO-MAN 2016 (IEEE International Symposium on Robot and Human Interactive Communication ), Associate Editor
  • ACM SIGGRAPH Asia Commitee Member (2008-2009)

Conference/Session Organizer

 Reviewer

  • Journals
    • IEEE Pervasive Computing, Special Issue – Conversational User Interfaces and Interactions, 2018
    • Journal of Behavioral Research Methods, 2015
    • International Journal of Affective Engineering, 2015
    • IEEE/ACM Transactions on Acoustic, Speech and Language (TASLP), 2014
  • Conferences
    • International Workshop on Spoken Dialogue System Technologies (IWSDS) 2017, 2018
    • Advanced Robotics, 2015
    • IEEE-RAS International Conference on Humanoid Robots (Humanoids), 2014
    • International Conference on Virtual Agents (IVA), 2014
    • Journal of Japanese Society for Artificial Intelligence (JSAI), 2012, 2013
    • International Conference on Social Robotics (ICSR), 2011

Patents

  • Conversational Robot (Japan 2010-221556)
  • Conversational Facilitation System and Robot (Japan 2008-304140)

Selected List of Research Intern Mentorship

Natural Language Understanding and Generation

  • Pranav Goel (Computer Science, International Institute of Information Technology), Deep Neural Network Based Conversational Strategy Classifier, Intern 2017 Summer … now with University of Maryland.
  • Wei Wang (Computer Science, Tsinghua University), Socially Conditioned Natural Language Generation using Recurrent Neural Networks: Intern 2017 Summer… now with Carnegie Mellon University.
  • Vivian Tsai (Computer Science, Johns Hopkins University), Incremental Socially Conditioned Natural Language Generation: Intern 2018 Summer.
  • Ying Shen (Computer Science, Fudan University), Natural Language Understanding, and Recurrent Neural Networks for Speech Markup Language (SSML) Generation: Intern 2016 Summer … now with Language Technologies Institute, Carnegie Mellon University.
  • Zhao Meng (Computer Science, Peking University), Natural Language Generation using OpenCCG: Intern 2016 Summer … now with ETH Zurich.
  • Akihiro Saito (Master of Computer Science, Waseda University), Automatic Expressive Opinion Sentence Generation for Enjoyable Conversational Systems: 2010 – 2012.

Dialogue Management

  • Zian Zhao (Shanghai Jiao Tong University), Socially-Conditioned Task Reasoning for a Virtual Tutoring Agent: Intern 2017 Summer.
  • Kweonwoo Jung (Mathematics, Carnegie Mellon University), User Simulation for Deep Reinforcement Learning Dialogue Manager of Socially Aware Conversational Agents: Independent Study 2016 Spring … now with Naver Corporation.
  • Yan Ting (Electronic Information Engineering, Beihang University), Statistical Dialogue State Tracking: Intern 2016 Summer    … now with Dartmouth University.
  • Xuchen You (Physics, Peking University), Inverse Reinforcement Leaning for Adaptive Social Reasoning: Intern 2016 Fall.
  • Iwao Akiba (Master of Computer Science, Waseda University), Facilitation Strategy Controlling Engagement Density As the Fourth Participant: 2012 – 2014 … now with Google, LLC.
  • Moemi Watanabe (Department of Computer Science, Waseda University), Partially Observable Markov Decision Process (POMDP) for Engagement Management: 2012 … now with NTT Data Corporation.
  • Azusa Todoroki (Department of Computer Science, Waseda University), Topic Detection and Management in Movie Recommendation Scenario: 2012 … now with Sony Corporation. 

Nonverbal Behavior Understanding

  • XuHai (Orson) Xu (Industrial Engineering, Tsinghua University), Automatic Listening Gesture Generation, and Computational Model of Rapport: Intern 2016 Summer.
  • Divya Sai (Computational Linguistics, International Institute of Information Technology), Automatic Head-nod Detection: Intern 2016 Summer.
  • Atsushi Ito (Master of Computer Science, Waseda University), Speaker and Addressee Classification using Nonverbal Cues in a Small Group: 2012 – 2014 … now with Yahoo Japan Corporation.

System Architecture Design and Integration

  • Vaibhav Nachankar (Master of Information Systems Management, Carnegie Mellon University), Development of Recommendation Engine and Wizard-of-Oz User Interface, Intern 2018 Spring.
  • Zihan (Atlas) Yu (Computer Engineering minor in Studio Arts, University of Pittsburgh), User Interface Infrastructure for Socially-Aware Conversational Assistant in Information Desk Scenario, Intern 2018 Spring.
  • Luo Yi Tan (Entertainment Technology Center, Carnegie Mellon University), Character Development in Unity3D : Graphic Programmer 2016 … now with Unity Technologies.
  • Anna Tan (Cognitive Science and Computer Science, Carnegie Mellon University), User Modeling for SARA at World Economic Forum: Independent Study 2016 Fall.
  • Sara Daraei (Applied Mathematics, Sharif University of Technology), Computational Architecture for Rapport-Aligning Peer Tutoring Agent: Intern 2015 Summer … Now with University of Pittsburgh.
  • Shan Han (Computer Science, Tsinghua University), System Implementation of InMind Agent User Study: Intern 2015 Summer … now with Carnegie Mellon University.
  • Emily Zhou (Computer Sciece and Art, Minor in Animation and Special Effecrs), User Interface Design and Development of InMind Agent, 2015.

Human-Agent / Human-Human Data Analysis

  • Deepika Mittal (Masters in Human Computer Interaction, Carnegie Mellon University), Discourse Analysis and Scenario Design of Information Desk Agent, Independent Study 2018 Spring.
  • Shuangni (Annie) Huang (Humanities & Arts in Decision Science and Fine Art, Additional Major in Human-computer Interaction, Carnegie Mellon University), Use Interface Design for Socially-Aware Conversational Agent in Information Desk Scenario, Intern 2018 Spring.
  • Neeti Ganjur (Science in Electrical & Computer Engineering and Engineering & Public Policy, Carnegie Mellon University), Developing Anonymization Pipeline of Human-Human Conversational Dataset, Intern 2018 Spring.
  • Jiayi (Cindy) Su (Information Systems, Carnegie Mellon University), Conversational Dataset Construction for Information Desk Scenario, Intern 2018 Spring.
  • Jingya Chen (Automobile Engineering, Tsinghua University), Field Study Analysis of a Socially Aware Robot Assistant at World Economic Forum,: Intern 2017 Summer … Now with Carnegie Mellon University.
  • Jiajia Li (Ph.D Visiting Scholar, Digital Media Art, Communication University of China), Designing Embodied Conversatioal Intelligent Personal Assistant: 2015 – 2017 … now with Qingdao University

Demos and Exhibitions

SARA Project (Carnegie Mellon University): Project Lead

SCHEMA Project (Waseda University): Project Lead

Media Coverage

SARA Project (Carnegie Mellon University): Project Lead

“Speaking with SARA certainly felt less jarring than talking to a regular chatbot. The system studies the words a person says during a conversation as well as the tone of his or her voice, also using several cameras to study the speaker’s facial expressions and head movements.”

MIT Technology Review

SCHEMA Project (Waseda University)