SmolVLA · Experiment 1: Training on 101 demonstrations

Why SmolVLA?

SmolVLA is a vision-language-action model of about 450 million parameters. It reads a text instruction, looks at the cameras, and predicts a chunk of 50 actions. I started from the pretrained base model and fine-tuned it on the 101-episode dataset. I wanted to know whether language and a pretrained vision backbone make up for having few demonstrations.

What settings did I use?

• Batch size 64, 10,000 steps. That is about 640,000 samples, around 10 passes over the 59,811 frames.
• Learning rate 1e-4 with 1,000 warmup steps, AdamW, gradient clipping at 10, and a cosine decay set to end at step 10,000. The default decays over 30,000 steps, which would have left the rate high at the end of a shorter run.
• Only the action expert and state projection train. The vision encoder stays frozen.
• Cameras renamed from side and front to camera1 and camera2, which is what the base model expects.
• One checkpoint, saved at the last step.

Why did it crash at step 800?

Around step 800 the run died with exit code 137. The cause: eight dataloader workers, each holding shared memory, pushed the container past its 50 GB memory limit. The machine reported 503 GB of RAM in total, which is the host, not what the container is allowed to use. I restarted with three workers. That made training slower (about 2 seconds per step) but stable, at about 30 GB of memory.

How did the run go?

6 hours 29 minutes of training. Loss: 0.047 at step 100, 0.018 at step 2,800, 0.014 at step 4,600, and between 0.010 and 0.013 over the final 2,000 steps. No nan values and no spikes. The pod ran from 22:32 UTC to about 06:38 UTC the next morning, about 8 hours 6 minutes, roughly $4.80.

What cost me time?

• No mid-run checkpoint. I set the save frequency equal to the number of steps, so there was exactly one checkpoint, at the end. A crash at step 9,000 would have lost everything. Save every few thousand steps, and copy only the last one to your own machine.
• Pods that disappeared. Two earlier pods vanished on their own while idle, and kept billing in the meantime. Check what is running on the account after every session.
• Library quirks on the pod. The video-decoding library torchcodec does not work there, so I removed it to let LeRobot fall back to PyAV. And pkill -f also kills the shell you are running it from, so kill by process ID.

What questions are still open?

• Was 10,000 steps enough? The loss was flat at the end, but flat training loss does not tell me whether the model generalises.
• Would a larger batch, or more steps, have changed the behaviour on the arm? I have one run, so I cannot say.
• Is the pretrained vision backbone helping at all with this scene? I did not test a run without it.

The short version

• Look at the container's memory limit, not the machine's RAM.
• Fewer dataloader workers cost speed, not quality.
• Decide the save frequency before launching a long run.

Next: SmolVLA Experiment 2, why a model that trained so well barely moved the arm.