Multimodal Learning from Demonstration

  • While recent advances in robotics research bring robots closer to entering our daily lives, real-world uses of autonomous robots are very limited. One of the main reasons for this is that designing robot controllers is still usually done by people specialized in programming robots: the lack of accessible methods for robot programming restricts the use of robots solely to people with programming skills. The motivation of this project is to provide algorithms that would enable non-expert users to design robot controllers for their specific needs, thus facilitating the integration of robots in people’s daily lives. Among humans, teaching various tasks is a complex process which relies on multiple means for interaction and learning, both on the part of the teacher and of the learner. Used together, these modalities lead to effective teaching and learning approaches, respectively. In the robotics domain, task teaching has been mostly addressed by using only one or very few of these interactions.
  • In this project we developed an approach for teaching robots that relies on the key features and the general approach people use when teaching each other: first give a demonstration, then allow the learner to refine the acquired capabilities by practicing under the teacher’s supervision, involving a small number of trials. Depending on the quality of the learned task, the teacher may either demonstrate it again or provide specific feedback during the learner’s practice trial for further refinement. Also, as people do during demonstrations, the teacher can provide simple instructions and informative cues, increasing the performance of learning. Thus, instructive demonstrations, generalization over multiple demonstrations and practice trials are essential features for a successful human-robot teaching approach.

Robot experiences the task during the demonstration [target visit and object delivery]

Robot performs task learned at abstract level, no trajectory following


Transfer of task knowledge from human to robot trainee


Transfer of task knowledge from robot to robot; former trainee robot acts as a teacher to another robot

Robot uses actions to communicate intentions [pick up an inaccessible object]

Robot uses actions to communicate intentions [traverse a blocked gate]

  • Design and Evaluation of Methods for Robot Learning by Demonstration, National Science Foundation, Early Career Development Award (CAREER), PI, Amount: $410,000, January 15, 2006 - January 14, 2011.