ACT · Experiment 3: 40 demonstrations (v1 + v2), stack and unstack

More demonstrations should help, so I recorded a second set of stack-and-unstack episodes and merged it with the first. This post is the data work and the training run. I have not tested this model on the arm yet.

What did the second recording look like?

It is the same task as the first recording (v1): 20 episodes of the arm stacking cup A on B and unstacking it again.

• Size: 11,141 frames. After trimming the idle frames with a 10-frame margin, 9,150 frames. The idle start dropped to 9.8 frames on average (at most 14), and both camera videos decode with all 9,150 frames.
• Things the checks flagged: the side camera looked shifted in episodes 0, 10 and 14, and the gripper's largest opening was only about 11 to 12 in episodes 3 and 13, against 15 to 47 in the others.
• What I did about it: I kept all 20 episodes. None of the flagged ones was dropped.

How were the two recordings merged?

With LeRobot's aggregate_datasets tool. The result has 40 episodes and 17,923 frames: 20 stack episodes (8,559 frames) and 20 unstack episodes (9,364 frames). Both camera videos decode with 17,923 frames each. I kept all 40 episodes.

What was wrong with the task text?

The second recording called its unstack episodes "Unstack A to B", and the first said "Unstack B to A", so the merged data had three task texts. I relabelled the first recording's unstack episodes to "Unstack A to B" with LeRobot's metadata tool. That changes the text only, not the video or the joint data. ACT ignores task text, but a model that reads instructions would not. The older copies of the first recording still say "Unstack B to A".

How was the model trained?

• Settings: the same as before: 20,000 steps, batch size 8, learning rate 1e-5, chunk size 100, 100 actions executed per chunk.
• GPU: an A40 at $0.59 per hour, because the RTX A6000 was out of stock.
• Speed and cost: about 2.35 steps per second, about 2 hours 20 minutes of training, and about 3 hours of pod time, because the first install took about 20 minutes (a slow dependency). About $1.80.
• Loss: 6.72 at step 200, 1.26 at step 2,000, 0.60 at step 3,600, 0.27 at step 6,400, 0.19 at step 9,200, 0.15 at step 12,000, 0.13 at step 14,800, 0.11 at step 17,600, and 0.106 at step 20,000. No nan values and no spikes.
• Checkpoint: I copied only the final one to my Mac (608 MB).

What did I learn?

• Give every recording the same task text before you merge them. Mismatched labels cost nothing for ACT and would hurt a model that reads them.
• Check each recording for camera shifts and odd gripper values before merging, and verify the frame counts after the merge.
• A slow install can take as long as a tenth of the training time. Start it with a log so you can see it is moving.

What questions are still open?

• Does the 40-demonstration model beat the 20-demonstration one? I have not run it on the arm.
• Would the camera shifts in the flagged episodes matter? I did not test that.
• Are two recordings made on different days really the same scene? I only checked the frames, not the lighting.

The short version

• Two recordings of the same task merged into 40 episodes and 17,923 frames, with the task text made consistent.
• One ACT run on them ended at a loss of 0.106, about $1.80, not yet tested on the arm.

Next: ACT Experiment 4, the same model on 101 demonstrations.