NOVA LINCS Receives Google Gift to research Drone Video AI Agent for Natural Disasters and Climate Change

Prof. João Magalhães, coordinator of the Multimodal Systems Group, received a Google Gift to research Vision and Language AI models to analyze videos obtained by Drones in scenarios of natural disasters and climate change.

Drone-mounted cameras opened a multitude of applications across a wide range of fields. They enable analysis over vast territories, facilitating comprehensive assessments of various landscapes and environments. Additionally, the ability to conduct temporal analysis enhances our understanding of temporal trends and changes, and the usage of aerial vehicles provides us the possibility of getting real-time information of vast territories.

This is particularly useful in the case of natural disasters and climate change where one needs to understand which damages occurred and may be a hazard for populations’ safety. These Multimodal AI systems should be easily accessible to the population in general and allow users to query such systems for visual evidence of roadblocks, flooding rivers, etc. Hence, our goal is to research models that can analyze videos obtained by Drones and allow users to ask questions about a video (VQA), search for particular video moments (video moment retrieval) and be informed in general about relevant events (video captioning).