How it works

A shared setup

Three SO-101 arms sit around a set of printed tracks, a cup, and a salt tray. Bandit picks the ball up from the tray and loads the upper track. Iris catches it in a cup and tips it onto a rocking section. Echo moves that section to return it to the tray.

The ball connects their separate tasks. One arm changes the next arm’s surroundings, and that change becomes a reason to move. You can inspect the whole arrangement in the interactive installation model.

Echo: a fixed gesture

Echo receives the ball in a predictable place. It has no camera and does not choose a new path. A recorded gesture moves the rocking track; a cue from Iris tells it when to start. Repetition is enough because the installation keeps this task consistent. Control is a sequence of joint targets rather than a new reach planned for each arrival. A change to the arrival position would require a change to the gesture or the setup.

Iris: a visual rule

Iris knows where the ball will arrive, but not when. A camera and a rule for an orange, round shape provide the missing cue. Once the ball appears, Iris catches and tips it with a familiar movement. The turn of its wrist then signals Echo. The visual rule provides the start condition; the catch-and-tip gesture remains scripted. This separates two parts of the task: detecting when the catch arrives and executing the familiar movement. Iris can respond to an arrival delay, but the setup still fixes the pickup location.

Bandit: a learned response

The salt tray presents a different problem: the ball can land in different places. A fixed reach could close on empty salt. We taught the pickup through demonstrations, connecting wrist and overhead images with the robot’s measured state and action commands.

A person guides a leader arm while the follower performs the movement. Each episode pairs the camera observations with six recorded motor-state channels and action commands. Those examples give a learned policy a way to predict how the reach should change with what the cameras see. The public demonstration dataset contains 115 episodes; a recording is a demonstration of the task, not a guarantee of autonomous success.

Recorded trajectories

Each demonstration records the six motor states over time. Using the robot’s joint geometry, we can reconstruct the tool position at every measured frame. The traces below compare twelve recorded reaches. The highlighted trace is episode 042, the same demonstration shown above.

From examples to actions

We trained ACT and diffusion policies with LeRobot. Both take camera images and robot state as inputs and predict a sequence of commands. ACT predicts an action chunk directly. Diffusion starts with a noisy candidate sequence and refines it over repeated steps. The components below use saved computations on demonstration observations, so we can inspect their predictions without presenting them as live inference.

ACT: predict a sequence

ACT uses a shared visual encoder for the two cameras. Image features and robot state enter a transformer that predicts a chunk of 100 actions. The model learns the relationship between an observation and the demonstrated commands. The released ACT checkpoint is the source for this recorded network view.

Diffusion: refine a sequence

The diffusion model encodes the cameras separately and combines those features with robot state. It starts with a noisy 64-step action plan and repeatedly refines it. The saved controller returns 32 actions at a time. This view uses computations from the released diffusion checkpoint.