Camera-guided skill: vision and inverse kinematics (no learning)

Why did the first attempt fail?

I asked for a skill that finds the cups with the camera and stacks them. What I first built was a replay: it checked with the camera that both cups were in their usual places, then played back one recorded demonstration at reduced speed. It failed twice, with both cups knocked over. A replay cannot react. If the cup sits a centimetre off in the gripper, the error goes straight into the placement.

How does the rebuilt skill work?

• Finding the cups. The front camera finds the paper sheet (the four extreme points of the white region) and the two cups on it. A homography maps pixels to table metres.
• Pixels to the robot's frame. I fitted a 2D similarity so the cups' standard spots land where the gripper actually grasped and released in 99 clean demonstrations. No calibration files.
• Inverse kinematics. The SO-101 model from a URDF file and the ikpy library. Position is solved exactly with the tool pointing down, tilted like the demonstrations. Each straight segment is solved in 1 cm steps and checked for accuracy and joint limits before the arm moves.
• Safety. A step cap per control tick, a scene check on four separate camera grabs that refuses to move if a cup is more than 4 cm from where it should be, and Ctrl+C that returns the arm to rest.

Why did the wrist-roll cap matter?

With the wrist roll capped at 90 degrees, the first IK run ended with cup A lying on its side. Raising the cap to 160 degrees, close to the roughly 155 the demonstrations use, fixed it. The numbers had been there all along: the recorded follower really reaches 163 degrees, and 65% of the demonstration frames are above 90. The original cap came from a servo overload at shutdown, and I had treated it as a hardware limit. It was a workaround for a different problem.

Why did the cup land on the rim?

Even with the right roll, cup A rested on the rim of cup B for a moment before the jaws opened, about 1.6 cm too close to the base. Two changes fixed it: aim the stacking point 1.6 cm further from the robot, and release 3 cm above the seated height so the cup drops in and the two cups centre themselves.

How do you center a cup with no markers?

I did not want stickers on the cups. At the hover over cup B the skill compares what both cameras see with a reference taken from a well-centred stack.

• A patch of the held cup (with the jaw holding it) is matched in the front and the side image. The side image works better as an edge map.
• Cup B never moves, so its position is measured once, at the start of the run, when nothing is in the way.
• The pixel error is converted to a robot-frame correction with a fixed table. I measured the table once, from two runs where I shifted the stacking point on purpose by 1.5 cm in x and then in y.
• The correction is applied once, limited to 2.5 cm, re-measured, and undone if the error grew. If the cup is already within about 3 mm, it does nothing.

In the last run it applied its first real correction (13 mm back and 3 mm sideways), and the pixel error fell from 32.5 to 13.8.

What did not work?

• A colour mask: the printed band on the cups is too pale.
• Matching cup B's band at the hover: the score drops to about 0.45 once the held cup hangs above it, so it is measured at the start instead.
• Probing the arm with small moves over cup B's rim: the matches were inconsistent, and it risked touching the rim.

What are the results?

With the wrist-roll cap at 160 degrees the skill stacked in 9 of 9 live runs. Two of those had the stacking point deliberately shifted by 1.5 cm and still stacked. Where I could measure the alignment without forcing an offset, cup A hung within about 1 to 2 mm of the reference in 3 of 4 runs. It assumes fixed cameras and a fixed sheet, and the cups must start near their usual spots.

What questions are still open?

• Does the centering help on average, or only on the occasional outlier? I have seen one real correction, so I cannot say yet.
• How far from the usual spots can the cups be before the mapping breaks? I only tested within about 4 cm.
• The sheet-size fit came out at 0.71 of the assumed A4 scale. Is the mapping accurate far from the cups? I did not check.

The short version

• Replaying one demonstration failed twice. Using the camera to decide where to go, and inverse kinematics to get there, worked.
• Check every limit against the data before you trust it. The 90-degree roll cap was the biggest single mistake.
• Two cameras and a reference view are enough to center a cup, with no stickers and no calibration files.

Next: every experiment in one table, and the lessons in one list.