The first ACT model, and a robot that would not move

What did I record?

The first recording was 20 episodes of the arm stacking cup A on B and unstacking it again, 11,413 frames in total with two cameras per frame. I used the Solo CLI's loop recording mode so the two tasks alternate without a manual reset between episodes.

Did the first training run work?

ACT, 20,000 steps, batch size 8, LeRobot's default learning rate of 1e-5 and chunk size of 100, on a rented RTX A6000 at $0.59 per hour. The loss fell from about 6.7 at step 200 to 0.106 at step 20,000, with no spikes and no nan values. That looked like a healthy run. It is also the only signal ACT training gives you in LeRobot: there is no validation split, so the only real test is putting the policy on the robot.

What happened on the robot?

• Run 1: the side camera had been moved since recording. The predicted actions stayed near the current pose and the arm did nothing useful. Moving a camera a few centimetres is enough to make a policy useless.
• Run 2: the camera was back where it had been. The arm moved but did not stack. It showed the same pattern again: it sat at the rest pose, then drifted.

Why did the arm stay at rest?

I analysed how each episode starts. The arm sat idle for an average of 105 frames, about 3.5 seconds, and up to 175 frames in the worst episode. With a chunk size of 100, ACT sees the rest pose followed by a hundred frames of nothing and learns that, from rest, the right thing to do is stay at rest. At run time the policy starts at rest, predicts rest, and never gets out of it.

What questions are still open?

• Was the idle time the only reason the arm stayed at rest, or would shorter action chunks have helped too? I did not test a chunk size below 100.
• How would this model have done after trimming and nothing else changed? (Post 3 answers part of that.)

The short version

• A good training curve says almost nothing about whether the policy works.
• Check the start of every episode before you train. Idle time at the start teaches "do nothing".
• Keep the cameras exactly where they were during recording.

Next: trimming the idle seconds, and every way a camera can fail.