SmolVLA · Experiment 2: First runs on the robot

Update, a few hours later: the biggest cause turned out to be the room lighting, not what is described below. SmolVLA Experiment 3 has the evidence. The wrist-roll cap and the instruction were real bugs, but the arm leaves rest without trouble once the light matches the training room, so the rest-start explanation further down is now doubtful.

What went wrong?

The fine-tuned model had a training loss of 0.013, yet on the arm it barely moved.

What were the two mistakes?

1. The wrong instruction. The model was trained on a single sentence, but the inference configuration still held an older one ("Stack Cup A to B and Unstack Cup A"). I had changed the instruction in the dataset and never in the inference settings. A text-conditioned model given text it has never seen is out of distribution.
2. The wrist-roll cap again. The 90-degree cap from ACT Experiment 2 was still active. 97% of the model's predicted rolls were above 90 (up to 123), while the arm's roll stayed at 89.6. The state the model saw never matched the demonstrations, and the arm stayed near the same pose for the whole run.

I fixed both and ran again. The arm still stayed close to the rest pose and crept: the shoulder lift moved from about -95 to -83 degrees in 49 seconds, where the demonstrations reach about -2 in around 5 seconds.

Is the model itself bad?

I ran the checkpoint on frames from a training episode, with no robot, and compared its predicted 50-action chunks with the real demonstration actions.

• Over a chunk, the average error was mostly 1 to 5 degrees per joint, with a few joints at 6 to 11. The next action was much closer. For example at the grasp: predicted pan 37, lift -2, elbow 35, roll 162, against 37, -2, 36 and 163 in the demonstration.
• The clear outlier was the very first frame, at the rest pose: chunk errors of 16 degrees at the shoulder lift and 26 degrees at the elbow.

So the model is good once it is moving. It fails at the start, because at rest the image looks the same whether the demonstration is about to move or is still idle. The 15 idle frames I left at the start of every trimmed episode are exactly that ambiguity. The model hedges, predicts almost nothing, and the arm never leaves rest. It is the same idle-start problem as the first ACT model, only smaller.

What do I try next?

• Start from a pose slightly out of rest (about 1.5 seconds into a demonstration), to see whether the model then does the whole task.
• Re-trim with a margin of about 3 frames so that the rest pose always means "start moving", then retrain.
• ACT on the same data (batch 8, 20,000 steps, running now at about 2.3 steps per second, about 2 hours 20 minutes and about $1.40) as a direct comparison with SmolVLA.

What questions are still open?

• Does starting out of rest make the SmolVLA model finish the task? I have not run that test.
• Does ACT trained on the same data have the same start-from-rest problem?
• Would a 3-frame margin remove the problem, or only reduce it?

The short version

• Test the checkpoint offline on its own training data before blaming the model. It separates "the model is bad" from "the model is being run wrong".
• Print the exact instruction the policy receives at inference, and compare it character by character with the training text.
• A cap that exists to protect a servo can silently change what the policy sees.

Next: SmolVLA Experiment 3, the fix turned out to be the room light.