Multimodal Systems collaborates with the Portuguese Navy in Multimodal Dialog AI Systems

NOVA researchers participated in the REPMUS exercise organized by the Portuguese Navy with the participation of NATO. The NOVA LINCS Multimodal Systems team created an AI assistant for the Navy to process complex tasks and critical information. The system was demonstrated to the Portuguese President of the Republic Marcelo Rebelo de Sousa and Admiral Gouveia e Melo, Chief of the Navy Armada, and the Minister of Defense Nuno Melo.

The team of researchers led by Prof. João Magalhães and Prof. David Semedo, co-authors of the first Large Language Model (LLM) trained in European Portuguese, have been collaborating with the Navy on the development of what we call the “silicon midshipman”, a prototype of an assistant capable of helping, through natural language , to solve small complex operational tasks described in official reference manuals.

[The team at REPMUS exercise (from left to right): First Lieutenant Vitor Rodrigues, Prof. David Semedo, Prof. João Magalhães, Eng. Iago Paulo, Eng. Rafael Ferreira, Eng. Daniel Pina, Eng. Inês Carvalho and First Lieutenant Ruben Carvalho]

This is the first implementation of the idea envisioned by Admiral Gouveia e Melo, Chief of the Naval Staff, where human-sailors and silicon-sailors will work in a symbiotic relationship. In this vision, human-sailors will be assigned actions related to ethics, based on values and philosophical principles that require a high level of self-awareness and accountability, or human relations, central to the art of command, resulting from their empathetic and emotional capacities, and unforeseen situations. Silicon-sailors will take on the repetitive, exhausting, dangerous, difficult, and support sailors in providing information related to complex tasks and critical information. 

During this exercise, the NOVA LINCS team was able to demonstrate a multimodal conversational AI system that answered questions grounded on the Navy publications and questions about video streams captured by drones.  

The exercise, with the silicon-sailor AI assistant, was an important landmark towards AI systems that work on Navy premises only and are grounded in Navy knowledge. It provides Navy with technology that is secure, auditable and effective, while preserving national sovereignty. 

Related news:

Carbon sailor and Silício sailor

Futuro da Marinha em ‘simulação’ num enorme exercício na costa portuguesa

Silicon Sailor: a Universidade Nova criou uma inteligência artificial para a Marinha Portuguesa

REPMUS – Robotic Experimentation and Prototyping with Maritime Unmanned Systems

EDA joins the Portuguese Navy Exercise REPMUS as co-organiser