· Summary
Every run in one table, and the lessons
What was tried, what each run cost, and what I would tell myself before starting.
What was tried, in one table
| Run | Model | Data | Result | Cost |
|---|---|---|---|---|
| 1 | ACT | 20 demos, untrimmed | loss 0.106, arm stayed at rest | not recorded |
| 2 | ACT | 20 demos, idle trimmed | loss 0.089, 1 clean stack in 2 tries | about $1.80 |
| 3 | ACT | 40 demos, trimmed | loss 0.106, not tested on the arm | about $1.80 |
| 4 | SmolVLA | 101 demos, trimmed | loss 0.013, stayed near rest in daylight, worked with the room light on (no success rate yet) | about $4.80 |
| 5 | Camera-guided skill | no training, anchors from the demos | 9 of 9 live stacks at roll cap 160, 1 of 1 failed at cap 90 | no GPU cost |
| 6 | ACT, batch 8, 20K steps | 101 demos, trimmed | loss 0.121, stacks from the usual cup positions, fails when the cups are shifted (no success rate yet) | about $1.80 |
What did I learn?
Data
- Look at the start and end of every episode before you train. Idle time teaches "do nothing".
- Trim, then verify that the new videos decode to the same frame counts as the data.
- Compute basic statistics: grasp positions, joint ranges, how much the scene varies. They explain failures later.
- Use one instruction for one task, and use the same text at run time.
- Vary the scene in the demonstrations. If cup B is always in the same place, the model never learns to look for it.
Training
- A falling loss is necessary and says very little. LeRobot's ACT has no validation split, so rollouts are the only test.
- Check the container's memory limit, and keep the dataloader workers modest.
- Save checkpoints during a long run, and copy only the one you need.
- Price the run before you start. ACT took about $1.80 per 20,000 steps, SmolVLA about $4.80 for 10,000 steps at batch 64.
- Check what is still running on the account when you finish.
Running a policy
- Keep the cameras exactly where they were when you recorded.
- Log camera brightness, state, predictions, sent actions and video. Diagnose from the logs.
- Treat every limit you add as part of the model's input. A cap on the wrist roll changes the state the policy sees.
- Test the checkpoint offline on training data to separate a bad model from a bad setup.
The classical route
- A replay cannot correct anything. Use the camera to decide where to go and the kinematics to get there.
- Fit your pixel-to-robot mapping to measurements you trust, here the demonstrations' grasp and release points.
- A reference view makes visual servoing possible without markers, and a forced offset gives you the camera-to-robot response without calibration files.
- Give every correction a safety path: limits, a re-measure, and an undo.
Process
- When a result surprises you, look for your own mistake first. Both big SmolVLA problems were mine.
- Write down the exact numbers (costs, steps, losses) as you go. I kept a running log, and every post here comes from it.
What questions are still open?
- Which approach wins on the same 5 to 10 test runs: ACT, SmolVLA or the skill?
- Is lighting the main thing holding the learned models back? It fixed SmolVLA once. I have not tested ACT, other lightings, or augmentation.
- How much of the skill's success is the fixed scene? I have not tested moved cups beyond about 4 cm.
- Does visual centering improve the average result, or only the rare bad grasp?
- How far can the cups move before SmolVLA and ACT fail, and how much varied data fixes both?
The short version
The fastest result came from the least learning: cameras, kinematics, and a mapping fitted to real demonstrations. The learned models need the exact instruction at run time and the lighting they trained in. The ACT run on the 101-episode data finishes soon, and the next post will compare it with SmolVLA and the skill.