Vivek Gopal Ramaswamy

SmolVLA · Learnings · Oct 24

Learnings from training SmolVLA

What I learned fine-tuning SmolVLA: a loss of 0.013, an arm that stayed at rest in daylight, and the cups it could not find.

· 2 min read · Vivek Gopal Ramaswamy

SmolVLA is a vision-language-action model of about 450 million parameters. It reads a text instruction and the cameras and predicts a chunk of 50 actions. I fine-tuned it once, on the 101-demonstration dataset, and then spent three experiments finding out why it did not work on the robot.

What was trained?

  • Model: SmolVLA, fine-tuned from the pretrained base model with the vision encoder frozen.
  • Data: the 101 demonstrations (59,811 frames after trimming) with one instruction.
  • Settings: batch size 64, 10,000 steps, learning rate 1e-4 with 1,000 warmup steps and a cosine decay to step 10,000.
  • Cost and time: 6 hours 29 minutes of training on a rented A40 at $0.59 per hour, about $4.80 for the pod. The loss went from 0.047 to 0.013.

What did I learn?

  • A low training loss told me nothing about the robot: 0.013, and the arm barely moved.
  • The instruction at run time must match the training text exactly. My inference settings still held an older sentence.
  • A wrist-roll cap changes the state the model sees. With the cap at 90 degrees the roll stayed at 89.6 while 97% of the model's predictions were above 90.
  • Lighting mattered most. The same checkpoint stayed near rest in daylight and worked with the room lamp on. On October 25 it stayed near rest again in natural light, while ACT, trained on the same data, worked in that light.
  • An offline test on training frames cannot show a lighting shift, because those frames carry the training lighting. Test with live frames too.
  • The data never moved the cups, so with cup B far away the model carried cup A to the old spot. What is missing is variety in the demonstrations, not more of the same.
  • Practical: check the container's memory limit (eight dataloader workers crashed the run), and save checkpoints during a long run.

What questions are still open?

  • What is the success rate with the lamp on, over 10 runs? I have not counted one.
  • How much change in light can the model tolerate, and would colour and brightness augmentation help?
  • Why does ACT cope with natural light when SmolVLA does not? I have guesses (ACT Experiment 5) but no test.
  • How far can the cups move before it fails?
  • Does recording the cups in many positions fix it, and does it fix ACT too? One dataset cannot separate data from model type.

The experiments

  1. 1

    SmolVLA · Experiment 1: Training on 101 demonstrations

    Batch 64, 10,000 steps, a memory crash at step 800, and about $4.80. · 3 min read    SmolVLA
  2. 2

    SmolVLA · Experiment 2: First runs on the robot

    A loss of 0.013, an arm that barely moved, and two mistakes that were mine. · 3 min read    SmolVLA
  3. 3

    SmolVLA · Experiment 3: Room lighting

    SmolVLA started working when I switched on a lamp. What changed in the pictures, and why my offline test missed it. · 4 min read    SmolVLA
  4. 4

    SmolVLA · Experiment 4: Cups moved

    With cup B moved far away, SmolVLA carried cup A to where cup B used to be. Why, and what data would fix it. · 3 min read    SmolVLA

Next: SmolVLA Experiment 1, fine-tuning on 101 demonstrations and the crash at step 800.

← All posts