Dataset · Oct 22 to 23
Dataset: 101 demonstrations
More data, one instruction, and a trimming job that was far too slow the first time.
Which datasets are there?
The data grew in four steps. All of them are the same scene: two paper cups on a white sheet.
| Dataset | Episodes | Frames | What it is |
|---|---|---|---|
| First recording (v1) | 20 | 11,413 | the arm stacks cup A on B and unstacks it again; 8,773 frames after trimming |
| Second recording (v2) | 20 | 11,141 | the same task recorded again; 9,150 frames after trimming |
| v1 and v2 merged | 40 | 17,923 (trimmed) | 20 stack episodes and 20 unstack episodes |
| 101 demonstrations | 101 | 90,662 | stack only, 30 seconds per episode, one instruction |
| 101, trimmed | 101 | 59,811 | idle frames removed with a 2-degree threshold and a 15-frame margin |
What did I record this time?
For the next round I dropped the unstack episodes and recorded only stacking: cup A onto cup B, 30 seconds per episode. Several recording sessions (86, 11 and 4 episodes) were merged into one dataset of 101 episodes and 90,662 frames.
What instruction did the model get?
Text-conditioned models read the task string, so I gave every episode the same one:
From the robot arm's perspective, pick up cup A on the right and gently stack it on top of cup B on the left, then release it and return the arm to rest.
It is 38 tokens, inside SmolVLA's limit of 48. It says "from the robot arm's perspective" because the front camera faces the robot, so the cup on the image's left is the robot's right cup. At run time the exact same sentence has to be typed. (Another thought to hold on to.)
Why did trimming take so long?
Trimming 101 episodes of two-camera video looked like a one-line command. It was not.
- First attempt: the tool wrote a PNG for every frame and then encoded with the software AV1 encoder. It would have taken hours.
- Second attempt: stream the frames straight into the encoder, use the Mac's hardware H.264 encoder, and split the episodes across four processes. About 12 seconds per episode, and the four shards merged in about 6 minutes.
The threshold for "the arm has started moving" was 2 degrees of joint motion, with 15 frames kept before and after. The result: 59,811 frames, 8.7 to 9.4 GB, uploaded to the Hugging Face Hub.
What did the data look like?
- Almost no variation. In all 101 demonstrations the cups sit within about 3 pixels of the same place. That is good for imitation learning and no use for testing generalisation.
- A very consistent grasp. Computing the gripper position with forward kinematics from the recorded joints, the grasp lands at the same point to within about 6 millimetres, and the release over cup B to within 4 to 9 millimetres.
- A big wrist roll. The wrist roll at the grasp averages 155 degrees, and 65% of all frames have a wrist roll above 90 degrees. I did not look at that number until much later.
What questions are still open?
- Is a 15-frame margin still too long? SmolVLA Experiment 2 first suggested it was, and SmolVLA Experiment 3 made that doubtful.
- With the cups in the same place in all 101 demonstrations, would any policy handle a cup that is 3 cm off?
- Does a second instruction, or none, change how a text-conditioned model behaves?
The short version
- Profile a data job on a few episodes before you run it on everything. Per-frame image files and software encoding are slow for video.
- One fixed instruction is fine for one task, but you must use the identical text at inference.
- Compute simple statistics on your data (grasp positions, joint ranges). They answer questions you have not thought to ask yet.
Next: Learnings from training ACT, then its five experiments.