How to Build a Dexterous Manipulation Dataset from Scratch

Dataset building is the unglamorous work behind every impressive robot demo. Do it well and the rest of the pipeline flies. Do it poorly and no model, no matter how large or clever, will save you. This is a practical checklist drawn from teams that have done this and shipped, with the mistakes they made along the way.

Start narrow. Not 'manipulation'. Not 'kitchen'. Something like 'slicing a cucumber into 5mm rounds on a wooden board with a chef's knife.' A tight envelope produces a coherent policy fast, which produces a working demo, which produces investor confidence and internal momentum. Broaden the envelope later, one axis at a time.

The most common failure at this stage is ambition. Teams try to capture 'all cooking' or 'all folding' and end up with a scattered dataset that trains a mediocre policy on everything and a good policy on nothing.

Decide up front which modalities you need: RGB from which viewpoints, depth, wrist camera, IMU, joint angles, force, audio. More is not always better. Every stream is a synchronization headache and a storage line item. A useful heuristic: include a stream only if you can articulate the failure mode that would result from omitting it.

Standardize the rig before you scale. If operator A captures at 60 Hz with two cameras and operator B captures at 30 Hz with three, you have two datasets, not one.

Cameras drift. IMUs drift. Gloves need per-session zeroing. Build a two-minute calibration routine that every session starts with, and log the resulting parameters in the session metadata. Six months in, when you want to normalize an old session against new hardware, you will have what you need.

Human demonstrators are the bottleneck and the largest recurring cost. Pay well, enough that the best people want to stay. Brief clearly with written task specs and short reference videos. Allow retakes without penalty, because a punishing pipeline produces gaming behavior that quietly poisons the dataset.

The variance between a bored operator and an engaged one shows up in policy quality within days. Small crew, well-paid, deeply engaged beats a large crew running through the motions every time. Retention is the metric, not headcount.

Every trajectory should pass a quality review before it enters the training set. The review checks: task completion, sensor coverage, absence of dropouts, absence of the operator's phone entering the frame, correct calibration. Build a small QA tool that plays back the trajectory with all sensors overlaid; a human can review a five-minute session in under thirty seconds.

Reject aggressively at the start. A dataset that is 90% clean beats a dataset that is 3x larger but 40% noisy. The noise poisons more than it teaches.

Datasets evolve. Track schema versions, calibration constants, operator IDs, environment IDs, hardware serial numbers, and firmware versions. Give every session a UUID and every recording a content hash. When someone asks 'why did the policy get worse after we added the March data', you will need every one of these to answer.

The same trajectory will probably be used by three different robots in the next two years. Store data in an embodiment-free format, joint angles in a standard skeleton, end-effector poses in the world frame, force in physical units, so that retargeting is a data-loader change and not a recapture.

The first thousand teach you about your pipeline, not about the task. Expect to throw away half of them once you learn what your rig actually captures. Budget accordingly. The second thousand is where the training-set signal starts, and the third thousand is where the policy starts looking like a product.