ACT · Experiment 2: Trimming idle frames (20 demonstrations)

How do I find idle frames?

I wrote a small tool with two commands. analyze reports, per episode, the idle frames at the start and end, the average joint speed, the gripper range, and whether a camera's first frame is far from the others (a sign it moved). trim writes a new dataset with the idle frames cut and a few frames of margin kept.

On the first dataset, 15 of 20 episodes had long idle starts. Trimming took it from 11,413 to 8,773 frames, and the idle start dropped to about 10 frames. I checked that both camera videos still decoded to exactly the same number of frames as the data.

Did trimming help?

Same settings as before (20,000 steps, batch 8) on the trimmed data: final loss 0.089, about two hours, about $1.80.

What else went wrong at the robot?

• Black cameras. In one run both camera frames were completely black (mean pixel value 0). The policy froze in one pose. After that, every run prints each camera's average brightness at the first step.
• A servo overload. The wrist-roll motor reported an overload error at shutdown. I capped the commanded wrist roll at 90 degrees, which stopped the errors. (Hold that thought: the cap comes back in the camera-guided skill and in SmolVLA Experiment 2.)
• Slow loop warnings. Every 100 steps the policy takes about 180 ms to compute a new chunk, which makes the control loop report a brief slowdown. The loop averaged 28 Hz, so this was harmless.

How did I debug a run?

I added a debug recorder to inference: the robot state, the predicted action, the action actually sent, timings, motor temperatures and loads, both camera videos, and a snapshot from each camera every 10 seconds. With it I could read a run frame by frame. One inference run had two attempts. The first failed: the cup was held above the other, dropped and toppled. The second stacked cleanly. One clean stack in two tries, with the release as the main failure.

What questions are still open?

• One clean stack in two tries is too few runs. What is the real success rate of the trimmed model over 5 to 10 runs?
• Why did the release fail in the first attempt of that run? I could see the cup drop, but I did not measure why.

The short version

• Trimming idle time moved the model from "never moves" to "sometimes stacks".
• Log the brightness of every camera at the start of a run.
• Record state, predictions, sent commands and video for every inference run. Most of the later diagnosis came from these logs.

Next: ACT Experiment 3, the same model on 40 demonstrations from two recordings.