Teaching a robot arm to stack cups

I wanted a robot arm to pick up one paper cup and set it on top of another, using only what its cameras see. More than the demo itself, I wanted to find out which approach gets there: a small imitation-learning policy, a larger model that follows a written instruction, or a classical pipeline with computer vision and inverse kinematics.

What is the setup?

• Arm: an SO-101 follower arm with six joints (shoulder pan, shoulder lift, elbow flex, wrist flex, wrist roll, gripper), driven in degrees. A matching leader arm is used for teleoperation when recording demonstrations.
• Cameras: two USB cameras at 1280 by 720 and 30 frames per second. The front camera looks at the robot head-on. The side camera looks across the table.
• Scene: two paper cups on a white sheet on a marble table. Cup A starts on the robot's right and cup B on its left. The task is to pick up A and set it on B. Across all demonstrations the cups sit within about 3 pixels of the same spot.

What software and compute did I use?

• Stack: LeRobot 0.4.5 and the Solo CLI for recording, training and running policies. Recording and inference ran on a Mac.
• Training: rented RunPod GPUs (an A40 at $0.59 per hour), with every run logged to Weights & Biases.
• Models tried: ACT (a transformer that predicts a chunk of 100 actions), SmolVLA (a 450-million-parameter vision-language-action model that reads a text instruction), and a hand-built skill that uses camera detection and inverse kinematics with no learning at all.

How did it go?

The first ACT model never left its rest pose. Trimming the idle seconds from the start of each demonstration fixed that, and a retrained model managed one clean stack in two tries. Scaling to 101 demonstrations gave a SmolVLA model with a very low training loss that crept near rest on the real arm in daylight and worked once I switched on the room light. ACT, trained on the same demonstrations, coped with natural light. The classical skill, once it was allowed to rotate the wrist as far as the demonstrations did, stacked in 9 of 9 live runs.

What questions are still open?

• Which approach is best on the same test: ACT, SmolVLA, or the camera-guided skill? I have not run them side by side yet.
• How much does the number of demonstrations matter once the data is clean?
• Do any of these still work if the cups start somewhere new?

The short version

• Three approaches: ACT, SmolVLA, and a camera-guided skill with no training.
• The skill is the only one that has stacked reliably (9 of 9 live runs). The SmolVLA model and a second ACT run are still being compared.
• The rest of the series is grouped by model: what I learned training ACT, what I learned training SmolVLA, and the camera-guided skill, each with its experiments, the numbers and the mistakes.

Next: the dataset behind every experiment, 101 demonstrations, and a trimming job that took far too long.