SmolVLA · Experiment 4: Cups moved

What did I try?

With the lamp on and the model working, I put cup B far away from its usual spot. The arm picked up cup A and carried it to the place where cup B always sat in the demonstrations. Cup B was somewhere else. The stack never happened.

A slight move of cup B still worked in my earlier tries. A big move does not. I have not measured where the limit is.

Why does it do that?

My reading is that the model never learned to look for cup B, because in the training data cup B never moved.

• In all 101 demonstrations both cups sit within about 3 pixels of the same place.
• The gripper closes on cup A at the same point to within about 6 millimetres, and opens over cup B at the same point to within 4 to 9 millimetres.

With data like that, "carry the cup to the usual spot" and "carry the cup to cup B" give the same action every time. The model can get a low training loss without ever reading where cup B is. Moving cup B is the first test that tells those two apart, and the model fails it.

I have not proved this. A model that ignores cup B and a model that sees cup B but has never seen it elsewhere would both fail the same way in this clip.

Is it because there is not enough data?

Partly, but not in the way I first thought. 101 episodes is a normal size for this kind of task. What is missing is variety: the number of different places cup B has been. With one position, 101 episodes carry the information of one.

What would I collect?

This is my plan, not something I have tested.

• Put cup B in 8 to 10 different places across the sheet, and a few cup A places too, with about 8 demonstrations each. That is 60 to 80 new episodes.
• Keep the room light the same as the lamp-on setup, or record in more than one lighting on purpose.
• Keep the existing 101 episodes, and drop episode 99, which is a dead recording.
• Trim the idle start to a few frames.

How does the camera-guided skill compare?

The hand-built skill finds cup B in the image on every run, so a moved cup is a normal input for it. I have only tested it with the cups within about 4 centimetres of their usual spots, so I cannot say how it does with cup B far away.

What questions are still open?

• How far can cup B move before SmolVLA fails? I have one far position and some small moves.
• Does the model react to cup A moving too?
• Does the ACT model trained on the same data fail the same way? (Update: yes, see ACT Experiment 4.)
• How many positions and how many episodes per position are enough? My numbers above are a guess.
• Can a model with a language instruction use "cup B on the left" to find the cup, if the data has varied positions?

The short version

• The model carried cup A to the usual place, not to cup B.
• The data had cup B in one place, so the model never had to look for it.
• The fix is more variety in cup positions, not more of the same episodes.

Next: the camera-guided skill, which uses vision and inverse kinematics instead of learning.