Vision + Force + Proprioception: The State of Multimodal Robot Learning

Multimodal robot policies claim to combine vision, force, proprioception, and sometimes audio into a single learned system. The word implies a rough equality between the modalities. In practice the equality is rare, and unpacking the actual role each modality plays is the first step to understanding why some multimodal policies work and others just have more input tensors.

Vision contributes global scene information: where things are, what kind of object is present, roughly what the scene wants. It generalizes well across environments and objects when the training distribution is diverse. It degrades gracefully under occlusion, lighting change, and background variation, provided the model has seen enough variation.

Force contributes physical event information: what is in contact, how hard, in what direction, and whether the contact is stable. It does not degrade under occlusion because the sensor is at the contact. It is the modality with by far the most information per byte around contact events and by far the least information per byte during free motion.

Proprioception contributes state: joint angles, velocities, torques. It is the modality most models take for granted, because it is cheap and precise, and it is what closes the loop between the action commanded and the action executed.

Audio contributes environmental context and, in a small but growing set of applications, contact classification. A screwdriver seating a screw sounds different from a screwdriver stripping one. Most policies ignore audio; a few use it explicitly, and the results suggest more will.

Early fusion concatenates modality-specific encodings before any joint reasoning happens. It is simple and it works when the modalities live in similar statistical spaces. It struggles when they do not, which is most of the time.

Late fusion runs modality-specific towers and combines their outputs near the action head. It preserves modality integrity but loses the cross-modal reasoning that makes fusion valuable in the first place.

Cross-attention fusion, the current default in serious systems, lets each modality attend to representations from the others across the depth of the network. It scales better than either simple approach and dominates the top of the leaderboards.

Tokenization approaches, treating force and proprioception as additional token streams alongside image patches and language, are the current experimental frontier. They are attractive because they slot cleanly into transformer stacks that already exist. They are unproven at scale for continuous modalities.

The clear advance is that vision-plus-proprioception has become a solved input format for a large class of tasks. Modern VLAs handle it well and the residual gains from architecture changes are small. The frontier has moved to what happens when contact-related modalities are added seriously.

The second advance is that alignment between modalities, the same event visible in the force stream and the video stream at the same wall-clock time, has become a first-class concern in dataset design rather than an afterthought. The teams building serious multimodal capture rigs treat sub-millisecond alignment as a hard requirement, and the resulting datasets support cross-modal training in ways that older datasets simply cannot.

The third advance is task-conditional modality weighting. Policies are learning to lean on force during contact-rich phases and vision during transport phases, rather than mixing all modalities uniformly across time. This is closer to how a human uses their senses and much more sample-efficient than the naive fusion recipes it replaces.

Force-heavy multimodal training. Most so-called multimodal work is vision-heavy with force as a garnish, for the data-availability reason discussed earlier in this series. Serious force-heavy training corpora are the missing ingredient for the next set of multimodal results, and the teams that produce them will define the ceiling of what deployed policies can do on contact-rich tasks.

The next piece looks at the specific way the current 'multimodal' language obscures how vision-dominated the actual state of practice remains.