Building a Multi-Modal Dataset: Video, IMU, Force, Audio

Anyone can record a single stream. Combining video, IMU, force, and audio into a coherent, time-aligned, retrievable dataset is where teams burn months. This is a field guide from the trenches, aimed at the engineer who has to actually make it work.

Every stream has its own clock. A camera runs on its internal oscillator. A wireless IMU runs on its own MCU. A glove reports over Bluetooth with variable latency. If you do nothing, these clocks will drift relative to each other and your ostensibly synchronous multi-modal dataset will actually be a set of streams that are close but not aligned.

The solutions in decreasing order of quality: hardware sync pulses on a shared wire, GPS-disciplined oscillators when hardware sync is impractical, software-side timestamp regression against a shared reference signal (a clap, an LED flash) at capture start and end. Whichever you pick, verify with a physical event and log the residual sync error in metadata.

Flat file per stream, indexed by session ID and timestamp. Resist the temptation to preprocess into one giant blob file. Downstream users will want different subsets, a language-conditioned model might not need force, an action-chunking policy might not need audio, and a per-stream layout lets them pull only what they need.

For very large datasets, chunked columnar formats (Parquet, Zarr) beat monolithic HDF5 for random access. Video is the exception; keep it as encoded MP4/MKV with frame-accurate index files.

Operator ID, environment ID, hardware serial numbers, firmware versions, calibration constants, ambient temperature, task variation notes. Missing metadata makes a dataset unusable in retrospect. A dataset without hardware serials cannot be filtered when a sensor recall reveals that certain units were drifting. A dataset without operator IDs cannot be de-biased if one operator was consistently faster than the rest.

The rule: if you might ever want to slice or filter on it, capture it now. Storage is cheap. Regret is not.

Video at 30 Hz. IMU at 200 Hz. Force at 200 Hz. Audio at 48 kHz. These streams have different natural rates, and forcing them all to a common rate destroys information. Store each at its native rate and let the training pipeline resample per model requirements.

The data loader is where the multi-modality actually happens. A well-written loader for this kind of dataset supports selective streams, on-the-fly resampling, time-window queries with configurable overlap, and augmentation hooks. Budget time to build this well. A slow loader will bottleneck training even on the biggest GPUs.

Before capturing ten thousand sessions, capture ten and run them end-to-end through a training pipeline. You will find bugs, a channel that is off by a factor of ten because of a units mismatch, a stream that is silently truncated at five minutes, a sync residual that grows with session length. Fix them at ten sessions, not ten thousand.