Dialog and vision are probably the most unique and complex human skills. They are mechanisms for humans to communicate knowledge, intentions, emotions, plans, etc. Yet, human-machine communication is still quite distant from the effectiveness and smoothness achieved by humans. Open-ended AI-based human-machine interaction, one where the topic is not defined in advance, requires understanding user intentions, answering in context, designing multi-step interactions, and, more importantly, maintaining a shared mental state. In this talk, we will delve into Open-Task Multimodal Conversational AI and share our vision towards assistants that glide through multimodal conversations to the successful completion of the selected goal. This is achieved through four key principles: a) robust dialog interaction, by making core dialog components (utterance processing, dialog manager, and response generator) tailored for an in-the-wild setting, b) effective and conversational task grounding, c) delivery of an immersive multimodal task guiding experience, and d) engagement maximization and user cognitive stimuli. The result is a novel multimodal curiosity-exploration conversational assistant, that balances user engagement and cognitive load, while guiding users through complex tasks.
30/11/2022 14:00
Multimodal Systems
Joao Magalhaes is an Associate Professor with Habilitation at the Computer Science Dep. at Universidade Nova de Lisboa. He holds a Ph.D. degree (2008) from Imperial College London, UK. His research aims to move vision and language AI closer to the way humans understand it and communicate. He has made contributions to the fields of multimedia search and summarization, multimodal conversational AI, data mining and multimodal information representation. He has coordinated and participated in several research projects (national, EU-FP7 and H2020) where he generalizes his research to many different application domains, e.g. BBC, VisionBox, Farfetch, Amazon. He is regularly involved in review panels, organization of international conferences and program committees. His work and the work of his group has been awarded, or nominated for, several honours and distinctions. He was the General Chair of the ECIR 2020, ACM Multimedia 2022 and the Honorary Chair for ACM Multimedia Asia 2021.
