PI: Fredrik Heintz, Linköping University
co-PI: Elin A. Topp, Lund University
The overall objective is to extend the capability of a single human operator to manage multiple robots carrying out missions in complex, dynamic, and unstructured environments by developing an agentic approach to mixed-initiative interaction using vision-language-action models. The Linköping PhD project will develop methods to effectively combine complex information gathered by the multi- robot system and vision-language-action models into an agentic system. The Lund PhD student will develop a multi-robot context tracking system that allows a single operator to use the agentic system to effectively control multiple robots during complex missions requiring rapid context switches.
The project directly relates to the Focus Area 1—Robots and Autonomous Vehicles of the ELLIIT 2030 Technology Foresight, and focuses mainly on the topics of distributed situational awareness and mixed–initiative interaction. It builds on results from the ELLIIT Call B project B09: Distributed situation awareness and mixed–initiative interaction for collaborative robotics, incorporating LLMs into the communication between multiple agents, specifically providing situation awareness to human operators when dealing with several robotic systems in complex scenarios. Given the combined Linköping – Lund (LiU–LU) expertise in robotics, knowledge representation and reasoning, machine learning, and human-robot-interaction, the team has the competencies required for the project and each PhD student will be supervised by one supervisor from each site. The project will also take advantage of the unique infrastructure at the two sites, especially the robot and UAV labs.
Project number: F7