ACT · Experiment 4: 101 demonstrations, stack only

What happened?

I trained ACT on the same 101 demonstrations (batch size 8, 20,000 steps, final loss 0.121) and ran it on the arm. When the cups sit where they sat in the demonstrations, it picks up cup A and stacks it on cup B. When I shift the cups, it does not work.

I have not counted a success rate for either condition. These are two clips and my observation, not a measurement.

Why does this matter?

Two very different models, a 450-million-parameter vision-language-action model and a small ACT policy, were trained on the same demonstrations. Both stack from the usual positions and both fail when the cups move. A limit that two different models share points at what they share, which is the data. In all 101 demonstrations the cups sit within about 3 pixels of the same place.

I cannot separate "the data is the problem" from "this kind of model is the problem" with one dataset. Both models saw the same pictures. A different dataset, with the cups in many places, is the test that would tell them apart.

Which model is better?

I do not know yet. At the usual positions both work, and I have not run them head to head. What I can say from my logs:

• Speed: ACT takes about 0.18 seconds to compute a new chunk of actions on the Mac. SmolVLA takes about 0.5 seconds.
• Cost: ACT took about 2 hours 25 minutes to train for roughly $1.80 of GPU time. SmolVLA took 6 hours 29 minutes for about $4.80.
• Lighting: SmolVLA failed in daylight and worked with the lamp on. ACT worked in natural light in the one clip I recorded (ACT Experiment 5), so the two models are not equally sensitive.

What questions are still open?

• What is the success rate of each model at the usual positions, over 10 runs?
• How far can the cups move before each model fails? Is the limit the same for ACT and SmolVLA?
• Does ACT also break in daylight? In one natural-light clip it did not (ACT Experiment 5).
• Does recording the cups in many positions fix both models, as I expect?
• Is the camera-guided skill the better choice for a task where the objects move?

The short version

• ACT works from the usual positions and fails when the cups move, the same as SmolVLA.
• Two models failing the same way on the same data points at the data.
• The next experiment is the same demonstrations plus 60 to 80 episodes with the cups in many places.

Next: ACT Experiment 5, ACT and SmolVLA in natural light.