Trimming idle frames, and every way a camera can fail

How do I find idle frames?

I wrote a small tool with two commands. analyze reports, per episode, the idle frames at the start and end, the average joint speed, the gripper range, and whether a camera's first frame is far from the others (a sign it moved). trim writes a new dataset with the idle frames cut and a few frames of margin kept.

On the first dataset, 15 of 20 episodes had long idle starts. Trimming took it from 11,413 to 8,773 frames, and the idle start dropped to about 10 frames. I checked that both camera videos still decoded to exactly the same number of frames as the data.

What did a second recording change?

A second batch of 20 episodes was trimmed the same way (9,150 frames) and merged with the first into a 40-episode dataset of 17,923 frames. The two batches labelled their unstack episodes differently, so I relabelled the task text in the merged dataset with LeRobot's metadata tool. ACT ignores task text, but a model that reads instructions would not.

Did trimming help?

Same settings as before (20,000 steps, batch 8) on the trimmed data: final loss 0.089, about two hours, about $1.80. The merged 40-episode model reached 0.106 and has not been tested on the arm.

What else went wrong at the robot?

• Black cameras. In one run both camera frames were completely black (mean pixel value 0). The policy froze in one pose. After that, every run prints each camera's average brightness at the first step.
• A servo overload. The wrist-roll motor reported an overload error at shutdown. I capped the commanded wrist roll at 90 degrees, which stopped the errors. (Hold that thought: the cap comes back in the next posts.)
• Slow loop warnings. Every 100 steps the policy takes about 180 ms to compute a new chunk, which makes the control loop report a brief slowdown. The loop averaged 28 Hz, so this was harmless.

How did I debug a run?

I added a debug recorder to inference: the robot state, the predicted action, the action actually sent, timings, motor temperatures and loads, both camera videos, and a snapshot from each camera every 10 seconds. With it I could read a run frame by frame. Run 7 had two attempts. The first failed: the cup was held above the other, dropped and toppled. The second stacked cleanly. One clean stack in two tries, with the release as the main failure.

What questions are still open?

• One clean stack in two tries is too few runs. What is the real success rate of the trimmed model over 5 to 10 runs?
• Does the merged 40-episode model beat the 20-episode one? It has not been run on the arm.
• Why did the release fail in the first attempt of run 7? I could see the cup drop, but I did not measure why.

The short version

• Trimming idle time moved the model from "never moves" to "sometimes stacks".
• Log the brightness of every camera at the start of a run.
• Record state, predictions, sent commands and video for every inference run. Most of the later diagnosis came from these logs.

Next: scaling to 101 demonstrations, and a trimming job that took far too long.