4D+ Motion Capture: Why Time-Aware Hand Data Beats Pose Snapshots

Motion capture has been around long enough that most people picture the film-industry version: reflective markers, an optical volume, an actor jumping around a stage while a solve reconstructs a skeleton. That pipeline is excellent at what it was built for, turning a human performance into a low-dimensional pose stream a rigged character can play back. It is a poor fit for training robots.

The reason is that dexterous behavior is not just where the hand was. It is when contact happened, how hard, in what direction, and how the hand responded in the next fifty milliseconds. A pose stream captures none of that. It gives you a beautifully clean 3D trajectory of a hand that is, for all a policy can tell, floating through empty space.

The industry term that has stuck for the alternative is 4D+ mocap. The name is worth unpacking because it hints at what makes the resulting data usable for physical AI.

Any competent mocap system captures X, Y, and Z position of every tracked joint. This is the geometric baseline. Whether the sensing is optical, inertial, magnetic, or hybrid, the output is a per-frame pose. For a full hand this is roughly twenty joints and the wrist, updated at least sixty times a second. Modern inertial gloves can push this to two hundred hertz cleanly.

Getting the geometry right is table stakes. It is also the easy part.

Time is what turns a pose stream into a trajectory, and time is where most datasets quietly go wrong. If the wrist IMU is sampled at 200 Hz and the finger sensors at 100 Hz and the camera at 30 Hz, you do not have 4D data. You have three separate time series glued together with interpolation, and any policy trained on it will learn to smooth over the exact micro-events, a first-contact spike, a slip recovery, a re-grip, that make dexterous behavior work.

True 4D capture means every channel is hardware-timestamped against a common clock, resampled deterministically, and delivered at a rate faster than the event you are trying to represent. Slip events happen on the order of tens of milliseconds. If your slowest channel is 30 Hz, you cannot see them. If your fastest channel is 30 Hz but your timestamps drift, you also cannot see them.

This is unglamorous engineering that separates data that trains a policy from data that looks like it should.

The plus in 4D+ is the set of signals that no external sensor can recover. Fingertip force. Contact area. Joint torque. Grip stiffness. In some capture rigs, EMG from the forearm. In others, ambient audio for the tap and scrape sounds that correlate with successful contact. These are the channels that let a downstream model learn a policy conditioned on contact rather than a policy conditioned on the appearance of contact.

The value shows up most clearly at task boundaries. When does a grasp start? When has an object been fully seated? When has a wipe covered the surface? All of these are trivial to answer with contact and force signals and nearly impossible to answer robustly from vision alone.

If you are collecting mocap for dexterous robotics, the checklist is short and severe. Every channel timestamped against one clock. Sampling rate at every joint fast enough to catch the transients you care about. Force somewhere in the stack, fingertip if you can, palm if you must. Contact events either sensed directly or reconstructed reliably in post. And a per-trajectory manifest that describes exactly what was captured, at what rate, and how it was aligned.

Data that meets this bar trains policies that behave correctly. Data that does not, does not, no matter how many trajectories you have. The teams building dexterity-first foundation models have learned to filter aggressively for the former.