Our Vision: General-Purpose Robots Driven by an LLM

We want a small, fast LLM to drive a robot of any form. The way to that vision runs through five capabilities and eleven breakthroughs. Nine of them can be started now, without waiting for chips.

The Vision and Its Scale

An LLM drives a robot of any form: one model under 100B parameters, and ideally under 30B, that completes the full loop of perceiving, understanding, predicting, commanding, and self-evaluating 20 times a second. It does not take the traditional VLA route, and at most mixes in a little of it. This route uses fewer resources than the traditional VLA route, and trains faster.

Unfolding the numbers in the vision gives the scale of the system we mean to build. In this article, one refresh is called a “beat.”

Quantity	Value	Calculation
Length of one beat	50 ms	1 second ÷ 20
Beats in an hour	72,000	20 × 3,600
Beats in a day of continuous running	1,728,000	20 × 86,400
Distance an object falls freely in one beat	1.2 cm	½ × 9.8 × 0.05²
Distance an object falls freely in four beats (200 ms)	19.6 cm	½ × 9.8 × 0.2²
Distance covered in one beat, walking at 1.5 m/s	7.5 cm	1.5 × 0.05
Distance an arm’s end moves in one beat at 1 m/s	5 cm	1 × 0.05
Body values read each second, taking 50 degrees of freedom as an example	3,000	50 × 3 (position, velocity, torque) × 20
Command values sent each second by the same robot	At least 1,000	50 × 20

Each number corresponds to one of our design choices:

The world changes within 50 ms, so our robot acts on prediction, and not only on reaction.
A day holds more than 1.7 million beats, so we build reliability and self-correction into every beat.
The more complex the body, the more values flow in and out, so we design a concise language for the robot.

Map of the Vision

Figure 1. One vision, five capabilities, eleven breakthroughs.

Reading from left to right: the vision is supported by five capabilities, and each capability is reached through two or three breakthroughs. The sensor language and the action language are the starting points. Once these two languages are fixed, the other nine have definite objects to work with.

Only breakthroughs 3 and 4 have to wait for progress in chips and compute. The other nine can be started now.

Capability 1 · Take It In: The World Arrives Every 50 ms in a Form the Model Can Read

Our model does not come to know the world by reading images. Space, trajectory, the data of every motor, and contact all arrive directly, in a form it can read. This rests on two breakthroughs.

Breakthrough 1: A Sensor Language

We will create a “sensor language” that writes continuous physical quantities as symbols an LLM reads directly. It is what lets us leave the VLM route with confidence. It has to be concise: with 50 degrees of freedom there are 3,000 body values a second, and a good language packs them into a very small symbol budget.

What we will explore:

How to represent space: the vocabulary and coordinates for objects, surfaces, free space, and people.
How to represent continuous quantities: the precision and the number of symbols for angle, velocity, torque, and time.
Who translates: the sensor itself, a small specialized module mixed in, or the model’s input layer.
How one language can describe a robot of any form and any combination of sensors.
Whether each beat writes everything, or only what changed.

What done looks like: for any robot, everything perceived in one beat is written within a fixed and very small symbol budget, and the spatial and body state the model reports from it matches the true values.

Breakthrough 2: An Always-Current World State and Memory

A robot lives through 72,000 beats in an hour. We give the model a world state of bounded size that is updated in place, so that after a full day it runs as lightly as in its first minute, and still remembers what matters.

What we will explore:

Where the current state lives: inside the model, in an external state table, or both.
How to compress and forget old information while keeping what the task needs, such as where the keys were put ten minutes ago.
How to store and retrieve memory across tasks and across days.
How to keep the organizing of memory outside the 50 ms main loop.

What done looks like: after hours of continuous running, the amount read per beat and the latency have not grown, and the model still answers questions about earlier states accurately.

Capability 2 · Think in Time: The Full Loop Is Computed Within 50 ms, Every Beat on Time

Perceiving, understanding, predicting, commanding, and self-evaluating are completed within the same beat, in one pass, and not split into a fast loop and a slow one. This rests on two breakthroughs.

Breakthrough 3: A Full Thought Within One Beat

We want the model to think through a whole loop in 50 ms. If tokens are generated one at a time, the speed target is set by the number of tokens per beat, and it is a single-stream, serial speed.

Tokens per beat for thinking and output	Single-stream speed target
10	200 tokens/s
100	2,000 tokens/s
1,000	20,000 tokens/s

There are two paths, and they can be used together: make a full loop need very few serial steps, and make each step as fast as it can be.

What we will explore:

The least computation a full loop needs.
Letting the thinking happen inside the model, and writing out only the conclusion.
Producing the output in parallel, all at once, and not one token after another.
Computing only the part that changed when a new state arrives.
Chips and architectures that reach this speed for models of 30B to 100B.

What done looks like: a model under 30B completes a full loop, including prediction and self-evaluation, within 50 ms, with decisions as good as when it thinks without a time limit.

Breakthrough 4: A Real-Time Runtime (Harness)

20 Hz is our commitment for every beat. We will build a runtime that runs continuously and keeps its state resident, so the model works as steadily as a heartbeat. It replaces calls made in the question-and-answer style.

What we will explore:

Aligning sensors, model, and motors to the same 50 ms beat.
Guaranteeing the upper bound and the jitter of latency, and not only its average.
When the compute is an external API, fitting the network round trip inside the 50 ms as well, and how the robot gets smoothly through a beat in which the network wavers.
Keeping state while the model runs for a long time, and updating it without stopping.

What done looks like: over a day of continuous running (1.728 million beats), every beat is on time. An occasional delay is absorbed smoothly, and the robot’s motion is unaffected.

Capability 3 · Think Right: The Model Knows Its Body, Sees Consequences, and Judges Itself Accurately

Understanding a task is already a strength of language models, and that is our starting advantage. On top of it, we make three breakthroughs.

Breakthrough 5: A Self-Model

Our model can drive a robot of any form because it knows its own body: how many degrees of freedom there are, how they are connected, how much force each motor can produce, and where the sensors are mounted. A new body needs no retraining, which is one source of our saving in resources.

What we will explore:

How the body tells the model about itself: by a “body description” the model reads and can use at once, or by the model moving and finding out.
How quickly, and with how little data, a new body can go from connected to usable.
How the model notices and adapts to changes in its body: load, wear, a different tool.
How much the model should know about the relation between a command and the motion that results.

What done looks like: a robot never seen before is connected and soon reaches a level of control close to that of a familiar robot, with no retraining.

Breakthrough 6: Prediction

Prediction is the core ability of our robot. A dropped object falls nearly 20 cm in four beats, and a walking robot advances 7.5 cm every beat. Our model knows what will happen before it happens, so it stays ahead of the world. Prediction runs in two directions: what its own actions will do to the world, and what events in the world will do to it.

What we will explore:

Whether prediction is qualitative (the cup will tip over) or quantitative (where it will be in 0.3 seconds), and how accurate each should be.
How far ahead to predict: the next beat, the next second, or the whole task.
How far the physical common sense a language model already has will carry, and how to supply the rest.
How the model expresses uncertainty, and chooses a safe action when it is uncertain.
Completing the prediction itself within 50 ms, and counting the beat that passes before a command takes effect.

What done looks like: the result the model expects before each action agrees closely with the actual result, and the rate of agreement can be measured. Faced with a sudden event in the world, the robot begins to move before the consequence appears.

Breakthrough 7: Self-Evaluation

Our model judges for itself whether it did well. On every beat it compares the actual state with the prediction it made a beat earlier. When the task ends, it judges whether the goal was reached. Trustworthy self-evaluation is the engine of self-correction and continued learning.

What we will explore:

How the standard for “done well” follows from the description of the task.
Keeping the judgment grounded in sensor facts at all times.
How a self-evaluation is used at once to correct the next beat, and how it is kept for later learning.
Calibration: when the model says it is 90% sure, it is right 90% of the time.

What done looks like: the model’s self-evaluation agrees closely with an independent outside judgment, and the confidence it reports matches its actual rate of success.

Capability 4 · Act Precisely: Commands Reach Every Motor, Steadily and Reliably

The model’s output becomes physical motion directly. This rests on two breakthroughs.

Breakthrough 8: An Action Language

We will create an “action language” in which the model gives every motor a command every 50 ms. An arm’s end moving at 1 m/s travels 5 cm in a beat, so one command describes how a motor behaves over the next 50 ms, and not just a single point.

What we will explore:

What a command contains: a target position, a velocity, a force, or “go there with this much compliance.”
How to write commands that are both short and exact: 50 degrees of freedom need at least 50 values per beat.
How commands are executed and joined between beats, including a contact that happens to fall between two beats.
How to guarantee that commands to several motors take effect at the same moment.
How one action language can serve arms, legs, wheels, and dexterous hands.

What done looks like: the same model, with the same action language, drives robots of different forms. The motion is smooth, and the force at contact is under control.

Breakthrough 9: Reliability

Our robot makes 1.728 million decisions a day. Our aim is that every one can be relied on: the rate of deviation is pushed extremely low, and an occasional deviation is corrected by the model itself on the next beat. For example, at 99.99999% correct per beat, a deviation appears about once in six days.

What we will explore:

How to keep the effect of any single deviation recoverable.
How the model finds and corrects, on the next beat, a deviation from the beat before.
Which limits are never crossed under any circumstances, and who guards them.
How to measure reliability this high.

What done looks like: over long continuous running the robot stays safe and stable, and this can be measured and shown.

Capability 5 · Learn Cheaply: Fewer Resources and Faster Training Than the VLA Route

The first four capabilities describe what the vision looks like. This one says why our route is better. It rests on two breakthroughs.

Breakthrough 10: A Training Recipe

Our advantage comes from one fact: most of the knowledge is already in the language model. We only have to teach it three interfaces, namely the sensor language, the action language, and the self-model, and fit all of the capability inside 30B.

What we will explore:

Sources of data that need no human teleoperation demonstrations: simulation, the robot’s own running records, and its own predictions and self-evaluations.
Which of the knowledge a language model already has can be used for control directly, and which has to be newly learned.
How a robot keeps learning while it runs: in context, in memory, or by updating weights.
How what one robot learns is passed to all robots, including ones of a different form.
How to fit the capability inside 30B while keeping the parts that matter.

What done looks like: the data, human effort, and time the model needs to learn a new ability or a new body can be counted exactly, and they fall as abilities accumulate.

Breakthrough 11: Proving “Cheaper and Faster”

We will prove that this route is better with a public ledger. With that ledger, the data decides which route is better.

What we will explore:

What to compare: the data, human hours, compute, and calendar time needed to reach the same task success rate.
Where to compare: the same robot, the same task, the same way of judging.
How the cost of pretraining the language model is counted.
Running cost: the inference spend per robot per hour.

What done looks like: on public, reproducible tasks, our route reaches a success rate equal to or higher than the VLA route, with clearly fewer total resources and in less time.