The fix was the room light

What happened?

After a day of chasing the SmolVLA model, I switched on the room's artificial light and ran it again. The arm left its rest pose, reached the cups and started stacking. I also moved the cup a little before a run, and it still worked.

Here are two clips I filmed on my phone.

I have not counted a success rate yet. These are two clips and one observation that a small cup move still works, not a measurement.

What did the logs show?

The same checkpoint, the same instruction and the same wrist-roll cap, in two different rooms.

• Daylight run: the shoulder lift stayed between -111 and -68 degrees for 49 seconds. It never left the area around the rest pose, and the biggest joint movement was 90 degrees.
• Lamp-on run: the lift went from -108 to -1 degrees, which is the height of the cups, and the biggest joint movement was 176 degrees.

What changed in the pictures?

In the daylight run the side camera saw a blown-out window and a washed-out white sheet. With the lamp on, the side view looks like the training data again.

The average pixel values tell a more mixed story, so I do not want to oversell them.

• Side camera: 130 in the training frame, 146 in the failing run, 121 in the working run.
• Front camera: 91, 95 and 100.

The front camera got a little further from training while the working run still worked. So brightness alone does not explain it. My guess is the direction and colour of the light, the shadows, and the window, all of which changed what the vision encoder sees. I have not tested that.

Why did my offline test miss it?

Two days earlier I ran the checkpoint on frames from the training episodes and found the predictions were close to the real actions. That test used the training pictures, so it used the training lighting. It could show that the model was good in the light it had seen. It could not show that the model breaks in a different light.

What was I wrong about?

I blamed the instruction, the wrist-roll cap, and the idle frames at the start of every episode. The first two were real bugs and I fixed them. The third now looks doubtful, because the arm leaves rest without trouble once the light matches. All three were worth fixing, but none of them was the main cause.

What would have caught it?

• Compare the live camera frames with the training frames before you run: brightness, exposure, and a look at the pictures side by side.
• Feed live frames, not training frames, into the offline test, and compare the predicted actions.
• Record the lighting in the dataset notes and keep the room the same, or record in more than one lighting.

My data-check tool has a camera comparison for exactly this, but I wrote it and never ran it. This is the case it was made for.

What questions are still open?

• How much change in light can SmolVLA tolerate? I only know two conditions.
• Does the ACT model from the same data break in daylight too? (It does fail when the cups move, like SmolVLA.)
• Would colour and brightness augmentation during training fix it, or would I need demonstrations in more than one lighting?
• What is the real success rate with the lamp on, over 10 runs?
• Did the 15 idle frames at the start matter at all?

The short version

• The SmolVLA model was not broken. It failed in a different light from the one it trained in.
• A test on training frames cannot show a visual shift. Test with live frames.
• Before you blame the model, compare what the cameras see now with what they saw when you recorded.

Next: what happens when cup B is moved far away.