LLM + Control in Real Applications¶
The earlier pages in this section worked on textbook problems — Laplace transforms, plot generation, a single PI tuning loop. Those are the right-size demos for how the pieces fit. This page is the other end: three real projects where an LLM sits somewhere in the control stack of a deployed robotic system. One drone swarm, one spacecraft, one manipulator. Each paper takes a different view of what the LLM is for — a planner, a controller, a reflective agent — and each diagram shows exactly where in the loop it lives.
Drone swarm choreography — SwarmGPT¶
Drone swarm performances are synchronized aerial dances set to music: beautiful to watch, hard to design. A human choreographer has to get both the artistic intent and the low-level collision / actuator constraints right. SwarmGPT splits those jobs: an LLM handles the artistic layer, and a deterministic safety filter guarantees the trajectory is actually flyable.
How the LLM fits
flowchart LR
U["User: song + style"] --> L[LLM choreographer]
L -->|waypoint trajectory| S[Safety filter]
S -->|collision-free refs| C[Onboard controller]
C --> D[Drone swarm]
S -.->|minimal corrections on constraint violation| LThe LLM proposes waypoint trajectories conditioned on the song and user intent. The safety filter makes minimal corrections — the smallest perturbations that restore feasibility — before the trajectory reaches the low-level controller. This is the tutorial's Tool Use pattern applied to robotics: the LLM is a creative proposer, a deterministic layer is the authority on what's physically allowed, and together they do what neither could alone.
Result
Validated on simulations up to 200 drones and real flights with 20 drones, synchronized to diverse music.
Autonomous spacecraft operations — KSPDG¶
The Kerbal Space Program Differential Games (KSPDG) is a public competition where teams build autonomous agents for non-cooperative spacecraft maneuvers — pursuit-evasion in orbit, for example. Classical approaches use optimal control or reinforcement learning. This work took a different bet: can an LLM, with the right prompt engineering and fine-tuning, operate a spacecraft directly?
How the LLM fits
flowchart LR
K[KSP simulator] -->|orbital state text| P[Prompt builder]
P --> L[Fine-tuned LLM]
L -->|RCS thrust commands| KThe LLM is inside the control loop, not supervising it. Each cycle, the current orbital state is formatted as a text prompt, passed to a fine-tuned model, and the model's JSON-formatted output — thrust commands on the reaction-control system — is applied back to the simulator. The control policy is the LLM's learned weights. Three techniques combine to make this work: prompt engineering (what to tell the model each cycle), few-shot prompting (example trajectories in context), and fine-tuning (adapting the base model to the thrust-decision task).
Result
Ranked 2nd in the KSPDG competition and is, to the authors' knowledge, the first LLM-based agent deployed for a space operations task.
Long-horizon robotic manipulation — Reflective Planning¶
Multi-stage manipulation — "assemble this part then screw it down then hand it over" — is where pretrained vision-language models (VLMs) should shine: they already know what the world looks like and what tools do. In practice they fail at long horizons because errors compound: a small misjudgment in step 2 becomes catastrophic by step 5. Reflective Planning addresses this not by training a bigger model but by adding a reflection loop at test time.
How the LLM fits
flowchart LR
G[Goal + RGB image] --> V[VLM planner]
V -->|proposed sub-goal| I[Imagined future state]
I --> R[Reflection critic]
R -->|revised plan| V
V -->|sub-goal| P[Diffusion policy]
P --> A[Robot arm]
A -->|new observation| VThe novel piece is the V → I → R → V loop before anything touches the arm. The VLM proposes a sub-goal, a generative model imagines what the world would look like after executing it, and a reflection step critiques the imagined outcome. This lets the planner catch its own compounding errors before they reach the robot, using only a pretrained VLM and a generative imagination model — no additional RL training, no MCTS.
Result
Outperforms state-of-the-art commercial VLMs and Monte Carlo Tree Search on long-horizon benchmark tasks.
The three roles at a glance¶
Each paper casts the LLM in a different part of the control stack — a useful taxonomy when you're thinking about where an LLM could live in your own system.
| Application | LLM role | Verification layer | How it relates to this tutorial |
|---|---|---|---|
| SwarmGPT (drone) | Supervisory planner — proposes trajectories from music + intent | External safety filter enforces feasibility | Tool Use pattern — LLM proposes, deterministic checker validates |
| KSPDG (spacecraft) | In-loop controller — fine-tuned model emits thrust commands directly | Baked into the learned weights via fine-tuning and few-shot examples | Same loop as PID Tuning, except the controller itself is the LLM, not a PI block |
| Reflective Planning (manipulation) | Reflective agent — imagines outcomes and critiques its own plan before acting | Self-supervised reflection against an imagined future state | Agent loops and Multi-agent — writer/checker, but with a generative imagination model as the critic |
The shape repeats: wherever an LLM is in the loop, something deterministic — a safety filter, a physics simulator, an imagined outcome — has to close the loop back on it. That's the same lesson from the PID Tuning page, just played out in three very different physical systems.
Next¶
Back to the LLM + Control overview.