SwarmChat: Enabling Human–Swarm Interaction and Robot Control via Natural Language

In today’s fast-moving world, coordinating teams of robots usually requires specialist knowledge and custom code. With SwarmChat, our EU-funded pilot under the UTTER programme, we’ve broken down that barrier—making it possible for anyone to direct and collaborate with swarms of robots using plain speech or text.

Why We Started

At Inventors Hub, we saw that foundational AI models excel at understanding language and even generating code, yet they’re seldom applied to swarm robotics. Traditional interfaces demand programming expertise and risk misinterpretation when instructions are ambiguous. We set out to build an interface that feels as natural as having a conversation, while embedding safety filters to catch hallucinations or unsafe commands before they reach the robots.

The Breakthrough Moment

Our turning point came when we layered three cutting-edge components into a single pipeline:

  • SeamlessM4T & EuroLLM for robust audio/text translation

  • LLaMA-Guard to filter out any unsafe or unclear intents

  • An LLM-driven behavior-tree generator to emit executable XML plans

This trio allowed us to take casual, unstructured user commands—like “spread out in a circle,” “tag the red marker,” or “return home”—and convert them on the fly into precise behavior trees that our Violet simulator can execute in real time.

Key Advantages of SwarmChat

  • Truly Conversational: No coding or robotics background required—just speak or type.

  • Safety-First: LLaMA-Guard catches and prevents risky commands.

  • Real-Time Execution: Watch XML behavior trees run instantly in our simulator.

  • Customizable Workflows: Easily fine-tune prompts or add new behaviors without rewriting code.

  • Scalable: Plug into any XML-driven swarm platform, from drones to ground vehicles.

Moving Forward

We’re now expanding SwarmChat beyond the simulator:

  • Field Trials with agricultural drone swarms later this quarter

  • Open-Source Release of our prompt-engineering toolkit

  • Live Workshops in partnership with UTTER partners to refine real-world interfaces

Our goal is to empower first responders, farmers, warehouse operators—and anyone—to harness multi-robot teams safely and intuitively.

Stay tuned here on the Inventors Hub blog for updates, demo videos, and early access invites. Ready to see SwarmChat in action? Try the live demo or request a pilot today!