Multiparty Conversation Facilitation Robots

Yoichi Matsuyama, 2015



In this dissertation, we study a framework for conversational robots facilitating multiparty conversations, which can maintain a group as a group, support group task achievements, and furthermore, entertain a group conversation itself. Starting with reviewing literature about theoretical frameworks of small group dynamics and participation structure, which have been discussed in fields of social psychology, linguistics and cognitive science, we then present a computational model of facilitation processes in multiparty conversations. The process mainly consists of procedural behavior selection regulating socially imbalanced situation and language generation for enjoyable conversations. The facilitation robot plays an unique role observing situations and taking initiatives to regulate equality of engagement density among participants. The procedural behavior production policy is optimized as a partially observable Markov decision process. The results of user studies conducted to evaluate the proposed procedures show evidences of their acceptability of robot’s behaviors and feeling of groupness perceived by participants. In the language generation process, we propose an automatic expressive opinion sentence generation mechanisms for enjoyable conversations. Expressed opinions are extracted from a large number of reviews on the web, and ranked in terms of contextual relevance, length of sentences, and amount of information represented by the frequency of adjectives. The sentence generator also has an additional phrasing skill. The results of user studies implied that mechanisms effectively promote interlocutors’ enjoyment and interests. As a robotic platform realizing the facilitation model above, we present the SCHEMA system, including its hardware design and network protocols interpreted among software modules. We also present the NANDOKU, a party game system for elderly care as an application of facilitation robots. Then we summarize the dissertation, and discuss future directions of multiparty conversation facilitation robots.

Table of Content


  1. Background and Purposeof the Dissertation
  2. Related Work
    1. Embodied Conversational Agents for Dyadic Interactions
    2. Multiparty Conversational Agents
  3. Research Objectives
  4. Dissertation Organization

Facilitation Framework

  1. Introduction
  2. Framework for Dyadic Conversation
  3. Framework for Multiparty Conversation
    1. Participation Role and Ratification
    2. Addressing and Recipient Design
    3. Engagement
  4. Layered Model of Conversational Processes and Protocols
    1. Facilitation Strategies
  5. Computational Architecture for Facilitation Robots
    1. Cognitive Architecture: Declarative and Procedural Memories
    2. SCHEMA Framework: Architecture for Facilitation Robots
  6. Conclusions

Engagement Density Control

  1. Introduction
  2. Theoretical Framework for Engagement Control
    1. Small Group Maintenance
    2. Engagement Density
    3. Procedures Obtaining Initiatives Controlling Engagement Density
    4. Adjacency Pairs : Timing of Initializing a Procedure
  3. Engagement Density Control Procedure Optimization as POMDP
    1. Partially Observable Markov Decision Process (POMDP) Basic
    2. Four-Participant Group Maintenance Mode
    3. Harmony Model
    4. Motivation Model
    5. Participants’ Action Mode
    6. System Action
    7. Belief State Update
  4. System Architecture
    1. Participation Role Recognition
    2. Motivation Estimation
    3. Adjacency Pairs Estimation
    4. Topic Management
    5. Question Generation
    6. Answer Generation
    7. Experimental Platform
  5. Experiment
    1. Preliminary Experiment
    2. Experimental Design
    3. Experiment 1: Appropriateness and Groupness by Usage of Procedures
    4. Experiment 2: Appropriateness of Timing of Initiating Procedure
    5. Results of Experiment 1 and 2
    6. Experiment 3: Evaluation of POMDP via User Simulation
  6. Conclusions and Future Work
    1. Summary and Contributions
    2. Extensions of POMDP
    3. Extensions of Situation Understanding

Language Generation

  1. Introduction
  2. Theoretical Framework of Language Generation for Enjoyment
    1. Small Talk
    2. Natural Language Generation Pipeline
    3. Opinion Mining and Sentiment Analysis
  3. Expressive Opinion Generation
    1. Document Collection
    2. Opinion Extraction
    3. Sentence Style Conversation
    4. Sentence Ranking
  4. System Architecture
    1. Natural Language Understanding Process
    2. Sentence Generation and Combination Process
    3. Factoid-typed Sentence Generation
  5. Experiments
    1. Experimental Design
    2. Experimental Platform
    3. Experiment 1: Acceptability of Sentence
    4. Results and Discussion of Experiment
    5. Experiment 2: Additional Phrasing
    6. Results and Discussion of Experiment
    7. Experiment 3: Comparison of Sentence Generation Algorithms
  6. Conclusions and Future Work
    1. Summary and Contributions
    2. Contextual Tracking
    3. Syntactic Structure Control
    4. Recommendation with Expressive Opinion
    5. Application to Other Domains

SCHEMA: Robotic Platform

  1. Introduction
  2. Exterior Design
  3. Mechanical Design
  4. Actuators and Electronics
  5. Sensor and Motor Module
    1. Speech Recognition
    2. Action Player
    3. Turret Control
    4. Speech Synthesis
  6. Network Middleware
    1. Existing Popular Middlewares: ROS, YARP, VHMs
    2. MONEA: Message-Oriented NEtworked-robot Architecture
  7. Discussions on Higher Level Protocols
    1. SAIBA : Multimodal Behavior Generation Framework
    2. Multi-Agent Simulator
  8. Conclusions and Future Work


  1. Introduction
  2. Nandoku: Elderly Care Application
    1. Robot as Communication Activator
    2. Group Communication Constraints
    3. Task Constraints
    4. Communication Activation Constraints
    5. Request-Answer Model
    6. Functions of Behaviors in Quiz Game Task
    7. Function of Behaviors in Communication Activation
  3. System Implementation
    1. Situation Understanding
    2. Behavior Evaluation
    3. Sentence Generation
    4. Content Design Support Tool
  4. Field Experiment
    1. Laboratory Experiment
    2. Experimental Design
  5. Results
  6. Conclusions and Future Work


  1. Summary of the Dissertation
  2. Significant Contributions
  3. Future Work