Learnings from training ACT

ACT is a small transformer policy that predicts a chunk of 100 actions at a time. I trained it four times: on 20 stack-and-unstack demonstrations, on the same 20 with the idle frames trimmed, on 40 demonstrations from two recordings, and on 101 stack-only demonstrations. I then tested the last model in natural light.

What was trained?

• Model: ACT with LeRobot's defaults (learning rate 1e-5, chunk size 100), trained from scratch.
• Data: 20 demonstrations (11,413 frames), the same 20 trimmed (8,773 frames), 40 demonstrations from two recordings (17,923 frames trimmed), and the 101-demonstration dataset (59,811 frames trimmed).
• Settings: 20,000 steps at batch size 8 every time.
• Cost and time: about $1.80 per run on a rented GPU at $0.59 per hour, where I recorded it, and roughly 2 to 2.5 hours of training each.

What did I learn?

• A falling loss says almost nothing about the robot. The first model reached a loss of 0.106 and the arm never left its rest pose.
• Idle frames at the start of each demonstration teach "do nothing". They averaged 105 frames, about 3.5 seconds. Trimming them took the model from never moving to one clean stack in two tries (loss 0.089).
• Keep the cameras where they were during recording, and log each camera's brightness at the start of a run. A moved camera made the first model useless, and one run had black frames.
• A cap on the wrist roll changes what the policy sees. The 90-degree cap I added after a servo overload turned out to be the wrong fix.
• When you merge recordings, make the task text consistent, check each recording for camera shifts, and verify the frame counts after the merge.
• With 101 demonstrations the ACT model (loss 0.121) stacks from the usual cup positions and fails when the cups are shifted. SmolVLA, trained on the same data, has the same limit, which points at the data. One dataset cannot separate data from model type.
• ACT worked in natural daylight although every demonstration was recorded under artificial light, while SmolVLA, trained on the same data, did not (one clip each, no success rate counted). Sensitivity to lighting differs between models, so test each one in the lighting you will use.
• ACT is cheap and fast: about $1.80 against about $4.80 for SmolVLA, and about 0.18 seconds per new chunk against about 0.5 seconds on the Mac, from my logs.

What questions are still open?

• What is the success rate at the usual positions, over 10 runs? I have not counted one.
• Does the 40-demonstration model beat the 20-demonstration one? I have not run it on the arm.
• How far can the cups move before ACT fails, and is the limit the same as for SmolVLA?
• Does ACT survive other lighting, such as dim light or a lamp from the side? It worked in one natural-light clip (ACT Experiment 5).
• Does recording the cups in many places fix it?

The experiments

1. 1

   ACT · Experiment 1: 20 demonstrations, stack and unstack

   20 demos, a clean training curve, and an arm that stayed at rest.Oct 20, 2026 · 2 min readACT→
2. 2

   ACT · Experiment 2: Trimming idle frames (20 demonstrations)

   A cleaning tool, a retrain, and a list of things that go wrong between a good checkpoint and a moving arm.Oct 21, 2026 · 2 min readACT→
3. 3

   ACT · Experiment 3: 40 demonstrations (v1 + v2), stack and unstack

   A second recording merged with the first: 40 stack-and-unstack demonstrations, one ACT training run, not yet tested on the arm.Oct 22, 2026 · 3 min readACT→
4. 4

   ACT · Experiment 4: 101 demonstrations, stack only

   The small ACT model stacks cups from the usual positions and fails when I shift them, just like SmolVLA.Oct 24, 2026 · 2 min readACT→
5. 5

   ACT · Experiment 5: Natural light

   ACT stacked cups in natural daylight even though every demonstration was recorded under artificial light. SmolVLA, trained on the same data, did not.Oct 25, 2026 · 3 min readACT→

Next: ACT Experiment 1, the first model, and why the arm would not move.