SwarmChat: Enabling Human–Swarm Interaction and Robot Control via Natural Language

SwarmChat is an intuitive, web-based interface that lets anyone command and interact with robot swarms using natural language. Funded by the European Union under the UTTER project, SwarmChat combines LLM models with behavior-tree generation to translate user instructions into precise, executable code—overcoming language ambiguity and AI hallucination for reliable, real-time human-swarm collaboration.

Project overview

SwarmChat is an EU-funded pilot under the UTTER programme that lets anyone, regardless of technical background, direct robot swarms using natural speech or text. It combines cutting-edge translation engines (Facebook’s SeamlessM4T and EuroLLM) with a safety filter powered by LLaMA-Guard and an LLM-based behavior-tree generator. User commands—spoken or typed—are converted into exact XML behavior trees and executed live in a Pygame swarm simulator, overcoming language ambiguity and model hallucinations. Backed by advanced prompt engineering, synthetic data fine-tuning, and an intuitive web interface, SwarmChat delivers a robust foundation for safe, real-time human-swarm collaboration in diverse application domains.

Project Gool

Enable effortless, reliable control of robot swarms by everyday users through natural language—speech or text—by translating their commands into executable behavior trees in real time, while ensuring safety and robustness against language ambiguity and LLM hallucination.

How It Works

When you speak or type a command into SwarmChat, our pipeline springs into action: first, SeamlessM4T or EuroLLM transcribes and translates your audio or text into canonical English. That text is then passed through LLaMA-Guard, which flags and filters out any ambiguous or unsafe intents. A dedicated LLM behavior-tree generator converts the vetted instruction into a precise XML tree, encoding actions and decision logic for each robot. Finally, the XML is dispatched to our Pygame-based simulator (or a connected swarm), where robots execute the plan in real time—all through a clean, browser-based interface that lets you watch the swarm move, adjust parameters, and issue follow-on commands with zero coding required.

Use Cases

SwarmChat’s natural-language control unlocks applications across industries. In search-and-rescue scenarios, operators can direct drone formations to fan out over a debris field or converge on detected heat signatures. Farmers can orchestrate swarms of ground vehicles or aerial drones to survey crop health, distribute seed or fertilizer, and adapt on the fly to field conditions. In logistics and warehousing, non-expert staff can task fleets of automated carts to pick, pack, and deliver goods simply by describing goals—“move pallets to dock A,” “sort boxes by weight,” or “clear aisle three.” From emergency response to smart agriculture to streamlined fulfillment, SwarmChat makes complex multi-robot coordination as easy as having a conversation.