Vivek Gopal Ramaswamy

ACT · Experiment 5 · Oct 25

ACT · Experiment 5: Natural light

ACT stacked cups in natural daylight even though every demonstration was recorded under artificial light. SmolVLA, trained on the same data, did not.

· 3 min read · Vivek Gopal Ramaswamy

Part of Learnings from training ACT

What happened?

Every one of the 101 demonstrations was recorded under the room's artificial light. On October 25 I ran the ACT model in natural daylight. It picked up cup A, stacked it on cup B and went back to its rest pose.

A few minutes later I ran SmolVLA in the same scene and the same daylight. It stayed folded near its rest pose for the whole clip, and the cups never moved.

ACT in natural daylight (24 s). The arm picks up cup A, stacks it on cup B and returns to rest.
A 12-second loop of the ACT run.
SmolVLA in the same daylight, a few minutes later (20 s). The arm stays folded near rest and the cups do not move.
A 12-second loop of the SmolVLA run.

I have one clip per model, in one scene, on one day. I have not counted a success rate for either, so this is an observation, not a measurement.

What is different between ACT and SmolVLA here?

Both models were trained on the same 101 demonstrations. What differs is the model, from my logs:

  • SmolVLA: fine-tuned from a pretrained model of about 450 million parameters. Its vision encoder stayed frozen, and it reads a text instruction. Training took 6 hours 29 minutes for about $4.80, and it takes about 0.5 seconds to compute a new chunk of actions on the Mac.
  • ACT: a small transformer trained from scratch, end to end, with a ResNet18 vision backbone and no text. Training took about 2 hours 25 minutes for about $1.80, and a new chunk takes about 0.18 seconds.

What do I think is going on?

These are my guesses. I have not tested any of them.

  • ACT learns its own image features from these demonstrations, while SmolVLA's vision encoder was frozen at the settings it had before fine-tuning.
  • The two models prepare the camera pictures differently. I did not check how.
  • ACT may lean more on the arm's own joint readings than on the pictures.
  • One clip per model is not a measurement, and something else about the two runs may differ.

Does ACT still fail when the cups move?

Yes. ACT Experiment 4 showed the same limit as SmolVLA: it works from the usual cup positions and fails when the cups are shifted. Coping with daylight does not mean ACT handles a changed scene in general.

What questions are still open?

  • What is the success rate of each model in natural light and under the lamp, over 10 runs?
  • Does ACT survive other lighting, such as dim light or a lamp from the side?
  • Would colour and brightness augmentation during training fix SmolVLA?
  • Which part of ACT explains it: training from scratch, the backbone, the unfrozen vision layers, or having no text?

The short version

  • ACT worked in natural light although it only ever saw artificial light. SmolVLA, trained on the same data, did not (one clip each).
  • Sensitivity to lighting differs between models, so test each model in the lighting you will use.
  • Coping with a new light is not the same as coping with moved cups: ACT still fails when they move.

Next: Learnings from training SmolVLA, then its four experiments.

↑ Learnings from training ACT  ·  All posts