Conversational AI and Robots

Course Objective

Did you ever wonder what it takes to develop a robot platform? Come learn and experience it in this course!

Robots are fun to work with! Especially social robots that can communicate with people. Robots are also very complex laboratories in which you can implement multimodal communication models, study how people communicate with robots and vice versa, and so much more.

In this course, you will:

1) Receive a general introduction to (embodied) Conversational AI;

2) Learn about our robot own robot platform: Leolani;

3) Be introduced to specific components of the Leolani platform, centred around its episodic knowledge graph, multimodality and their underlying data structures.

4) Develop a solution to a complex multimodal communication problem in a virtual-world context;

5) Learn how to assess and evaluate conversational agents.

Course Content

You will learn about the architecture of a communicating robot and its different components, starting from the sensors that pick up signals (audio and vision), the language understanding and generation models and the brain in which the robot stores all interpretations of the encounters with people. You will read overview papers that explain the state-of-the-art of human-robot communication and you will interact with the robot and report on the findings. You will gain a deep insight in what it takes to build a communicative robot, what the difficulties are and how to evaluate multimodal interactions.

Within the Leolani project, we study the relation between the world and how people and robots make reference to the world. The central theme of this course is the relation between identity, reference and perspective. What things do we identify in a shared space or situation, how do we refer to these things and what is the perspective of the communicating partners when referring in different ways.

All students will interact with the robot platform and evaluate your interactions in different ways.

You will write an individual report on the interaction and the evaluations. In addition, you work in small teams on a student project related to our research.

Additional Information Teaching Methods

There are two classes per week, each two hours. There will be regular classes in which we discuss literature, explain the robot models and explain the major concepts. There will also be hands-on sessions during which you work in small groups with the robot platform and work on your group project.

Method of Assessment

The assessment has two components:

1) You will write an individual report on your interactions and evaluation of the robot platform;

2) You will write a report describing the implementation and results of your group project.

Additional Information Target Audience

Students Research Master Humanities, Track Linguistics - Human Language Technology; students Master Artificial Intelligence.

Students must have programming experience – you can’t complete the course without it. We expect all students to be able to program in Python.

Additional Information

The course will encourage students to use (Generative) AI during the development of the group project (i.e., complex multimodal communication). AI is, however, not allowed to derive analyses from the data collected for both assignments. All content presented in the reports should be created by students. The use of AI in the written reports is strictly restricted to language revision and reformulation.

Recommended background knowledge

Knowledge of LLMs, Knowledge Graphs and NLP are all welcome.