Summary · Oct 24
Summary: every experiment in one table, and the lessons
What was tried, what each run cost, and what I would tell myself before starting.
What was tried, in one table
| Experiment | Model | Data | Result | Cost |
|---|---|---|---|---|
| ACT 1 | ACT | 20 demos, stack and unstack, untrimmed | loss 0.106, arm stayed at rest | not recorded |
| ACT 2 | ACT | the same 20 demos, idle frames trimmed | loss 0.089, 1 clean stack in 2 tries | about $1.80 |
| ACT 3 | ACT | 40 demos (two recordings merged), trimmed | loss 0.106, not tested on the arm | about $1.80 |
| ACT 4 | ACT | 101 demos, stack only, trimmed | loss 0.121, stacks from the usual cup positions, fails when the cups are shifted (no success rate yet) | about $1.80 |
| ACT 5 | ACT on the robot | the model from ACT 4 | works in natural daylight although all demos were recorded under artificial light (one clip, no success rate) | no extra training cost |
| SmolVLA 1 | SmolVLA | 101 demos, trimmed | loss 0.013 | about $4.80 |
| SmolVLA 2 | SmolVLA on the robot | the model from SmolVLA 1 | in daylight it stayed near rest; two inference-side bugs (the instruction text and the wrist-roll cap) were fixed first | no extra training cost |
| SmolVLA 3 | SmolVLA on the robot | the same model | with the room light on it works (no success rate counted); in natural light on Oct 25 it stayed near rest again | no extra training cost |
| SmolVLA 4 | SmolVLA on the robot | the same model | with the cups moved far away it carries the cup to the old spot | no extra training cost |
| Skill | Camera-guided skill | no training, anchors from the demos | 9 of 9 live stacks at roll cap 160, 1 of 1 failed at cap 90 | no GPU cost |
What did I learn?
Data
- Look at the start and end of every episode before you train. Idle time teaches "do nothing".
- Trim, then verify that the new videos decode to the same frame counts as the data.
- Compute basic statistics: grasp positions, joint ranges, how much the scene varies. They explain failures later.
- Use one instruction for one task, and use the same text at run time.
- Vary the scene in the demonstrations. If cup B is always in the same place, the model never learns to look for it.
Training
- A falling loss is necessary and says very little. LeRobot's ACT has no validation split, so rollouts are the only test.
- Check the container's memory limit, and keep the dataloader workers modest.
- Save checkpoints during a long run, and copy only the one you need.
- Price the run before you start. ACT took about $1.80 per 20,000 steps, SmolVLA about $4.80 for 10,000 steps at batch 64.
- Check what is still running on the account when you finish.
Running a policy
- Keep the cameras exactly where they were when you recorded.
- Log camera brightness, state, predictions, sent actions and video. Diagnose from the logs.
- Test every model in the lighting you will use. Sensitivity differs between models: SmolVLA failed in natural light and ACT, trained on the same demonstrations, did not.
- Treat every limit you add as part of the model's input. A cap on the wrist roll changes the state the policy sees.
- Test the checkpoint offline on training data to separate a bad model from a bad setup.
The classical route
- A replay cannot correct anything. Use the camera to decide where to go and the kinematics to get there.
- Fit your pixel-to-robot mapping to measurements you trust, here the demonstrations' grasp and release points.
- A reference view makes visual servoing possible without markers, and a forced offset gives you the camera-to-robot response without calibration files.
- Give every correction a safety path: limits, a re-measure, and an undo.
Process
- When a result surprises you, look for your own mistake first. Both big SmolVLA problems were mine.
- Write down the exact numbers (costs, steps, losses) as you go. I kept a running log, and every post here comes from it.
What questions are still open?
- Which approach wins on the same 5 to 10 test runs: ACT, SmolVLA or the skill?
- Why does lighting break SmolVLA but not ACT? In one clip ACT worked in natural light. I have not tested other lightings, more runs, or augmentation.
- How much of the skill's success is the fixed scene? I have not tested moved cups beyond about 4 cm.
- Does visual centering improve the average result, or only the rare bad grasp?
- How far can the cups move before SmolVLA and ACT fail, and how much varied data fixes both?
The short version
The fastest result came from the least learning: cameras, kinematics, and a mapping fitted to real demonstrations. The learned models need the exact instruction at run time, and SmolVLA also needs the lighting it trained in (ACT coped with natural light in one clip). ACT trained on the same 101 demonstrations (ACT Experiment 4) has the same limit as SmolVLA: it works from the usual cup positions and fails when the cups move. The next experiment is more varied demonstrations.