ChatGPT Enables Roboticist to Choreograph Flying Robots

Professor Angela Schoellig from the Technical University of Munich (TUM) uses ChatGPT to develop choreographies for swarms of drones to perform along to music. An additional safety filter prevents mid-air collisions. The researchers’ results demonstrate that large language models (LLMs) such as ChatGPT can be used in robotics.

The web interface is easy to use: Doctoral student Martin Schuck selects a music track and enters a text requesting a suggested choreography. Additional instructions can be given to the drone swarm via another prompt in the ChatGPT tool before an algorithm check whether the suggested flight paths are feasible.

The screen in the Learning Systems and Robotics Lab now shows a simulated airfield with six drones flying in circles to the music. If the scientist from the Chair of Safety, Performance and Reliability for Learning Systems at TUM likes this choreography, he logs it in. Soon afterward, six palm-sized drones take off from the floor of the robotics lab.

In Schoellig’s laboratory, the research team has installed six ceiling-mounted cameras in a room measuring around 40 square meters and three meters high. Crosses are marked on the floor with insulating tape. These indicate the starting positions of the drones. Once the computer has verified a possible choreography, they can take off.

The cameras detect the position of the quadrocopters, which are equipped with four propellers and motors, 200 times a second. The system compares them with the desired position. The “airshows” in the Learning Systems and Robotics Lab, which the research team realizes with up to nine drones, are 100 percent safe today. Without the special safety filter, only one in four demonstrations is accident-free.

Schoellig combined ChatGPT with the safety filter for the “Dance of the Flying Robots.” According to her, the ChatGPT AI tool was primarily created to generate texts, but it can also suggest choreographies. “However, it initially knows nothing about the properties of drones and physical limits for the flight paths. So, it is clear that ChatGPT makes mistakes,” she added.

The additional safety algorithm closes this gap by mapping out flight paths for the proposed choreography so precisely that mid-air collisions are completely avoided. Drones can even approach each other diagonally. Schoellig calls the overall concept of ChatGPT and security filter designed to use several flying robots “SwarmGPT.” The tool generates the processes in the air and at the same time serves as an interface between the robot and the human, who does not require any expert knowledge.

When Schoellig began her drone research almost 15 years ago, choreographies were hand-crafted. It took more than three years to develop the first six choreographies for six drones and get them up and running. “ChatGPT has brought about a quantum leap,” she said. Over the past three months, the researchers have experimented with over 30 choreographies for up to nine drones. Today, it takes the researchers only around five minutes to develop a safe choreography for 30-second music clips with three drones. The more drones are added, the more time ChatGPT needs for its calculations and the longer it takes to propose a choreography. But Schoellig is certain: “The concept is scalable.”

Can other robots be used with a similar interface via ChatGPT? For robots that use voice control to pick up objects, lay cables or open doors, the success rate of those actions is currently only 63, 56, and 80 percent. So far, applications in other robotics scenarios have been somewhat unreliable. For Schoellig, this is an incentive, “I assume that our approach will keep getting better in other scenarios, too.” It may soon be possible to reprogram suction-based and industrial robots simply by voice command without the need for expert knowledge or programming skills.

For more information, contact Andreas Schmitz at This email address is being protected from spambots. You need JavaScript enabled to view it.; +49 162-27-46-193.