Article details

Research area
Natural language & AI

New York, NY, USA


Heriberto Cuayahuitl, Lutz Frommberger, Nina Dethlefs, Antoine Raux, Mathew Marge, Hendrik Zender

Introduction to the Special Issue on Machine Learning for Multiple Modalities in Interactive Systems and Robots


ACM Transactions on Interactive Intelligent Systems (TiiS) – Special Issue on Multiple Modalities in Interactive Systems and Robots. Vol. 4, Issue 3. ACM.

This special issue highlights research articles that apply machine learning to robots and other systems that interact with users through more than one modality, such as speech, gestures, and vision. For example, a robot may coordinate its speech with its actions, taking into account (audio-)visual feedback during their execution. Machine learning provides interactive systems with opportunities to improve performance not only of individual components but also of the system as a whole. However, machine learning methods that encompass multiple modalities of an interactive system are still relatively hard to find. The articles in this special issue represent examples that contribute to filling this gap.

Read/download now