MULTIMODAL SYSTEMS

PI: Nuno Correia

Multimodal Systems research is focused in methods and tools “to create, access, and interact all forms of digital content in any device or scenario”. We will also work on algorithms to “model, analyse, and visualise vast amounts of data from which to extract more value, to make an intelligent use of data coming from different sources”. Thus, the Multimodal Systems Group is fully aligned with the EU Horizon 2020 ICT research topics for “Content technologies and information management”.The multimodal systems group targets three principled research areas in the area of digital content information access and interaction:

  • Multimodal and natural computer interaction: This line of research pursues novel interaction mechanisms based on computer vision (e.g., gestures), surface devices and other mechanical devices to augment user interaction (e.g., leap motion). We will also research methods to assess user behavior to enrich the feedback provided to users.

  • Machine translation: is probably the key area for making existing worldwide linguistic barriers to vanish, and thus enable a true globalization in all respects. Building on past success and the creation of a spin-off, this line of research will tackle new languages (Mandarin, Arab, Japanese, Russian and Latin), while enabling portable applications involving translation, and requiring technologies as OCR of handwriting, and speech recognition and speech synthesis.

  • Multimedia information retrieval: this line of research will target search over diverse information sources and the scalability of large-scale multimedia information retrieval systems. More specifically, we will research the information analysis algorithms over multiple information sources, and algorithms to effectively index multimedia information.

A fourth major activity in our group will target technology transfer processes and a strong connection to society. More specifically:

  • Tools for healthcare, creative and media industry: This line of research is focused in bridging the research conducted in the above topics with real world problems. This interaction goes both ways: the principled methods, tools and algorithms are turned into prototypes to provide solutions, and the research for novel principled methods are also inspired by problems arising from this interaction with the society.

We consider this fourth activity to be fundamental to strengthen our position as a leading national research group and a technology provider to enable services based on digital content and data. For the next period we aim at becoming a key international research group in this area, by leveraging on the “EU cultural and creative sectors that accounts for 3.3% of GDP and employ 6.7 million people (3 % of total employment).

The group publishes at major conferences and journals such as ACM Multimedia, IFIP Interact, ACM CIVR, ACM CHI, ACE, MobileHCI, ACM ITS, IFIP Interact, Computer & Graphics, Multimedia Tools and Applications, ACM Transactions on Information Systems and Multimedia Systems journal.