Vision-Tactile Sensors Are Beautiful. They're Almost Never in Real Datasets.

Walk through any robotic manipulation paper from the last three years and you will find beautiful figures of GelSight or DIGIT streams: heat maps of contact, arrows of estimated shear, colored overlays of slip events. Then walk into the training corpus that model was trained on and count how many demonstrations actually contain a tactile stream. The number is almost always small. Often it is zero.

This gap between what vision-tactile sensors can measure in a benchmark and what they actually contribute to a production robot policy is one of the more honest anomalies in the field. It is worth naming.

Open X-Embodiment, the largest public cross-embodiment manipulation corpus, contains roughly a million trajectories aggregated from dozens of labs. The fraction of those trajectories that include any tactile stream at all is in the low single digits. The fraction that include a synchronized, calibrated, high-rate tactile stream that a downstream policy could actually condition on is smaller than that.

Inside foundation-model efforts the ratio is not much better. π0 and its successors were trained overwhelmingly on vision-plus-proprioception. Diffusion Policy variants published across the community are almost entirely vision-driven. The specific ablations that would show tactile pulling its weight in a large training run are largely absent, because the data to run those ablations at scale does not exist.

Meanwhile the number of papers demonstrating that tactile helps on a specific task keeps growing. Insertion. Cable routing. Fruit picking. Cloth manipulation. In every case where someone bothers to instrument a fingertip and collect a focused dataset, tactile earns its keep. And in every case the resulting dataset is small enough that it never leaves the paper.

The straightforward reason is throughput. A capture rig with a GelSight fingertip on each of two grippers is a delicate instrument. It needs calibration before each session. Its elastomers wear and drift. Its cameras are one more source of bandwidth and dropped frames. A lab producing a hundred demonstrations for a paper can absorb that overhead. A capture operation producing ten thousand demonstrations per week cannot.

The subtler reason is embodiment. Most vision-tactile fingertips do not fit on the hands that people actually want to deploy. Fitting a GelSight onto a Franka gripper is straightforward; fitting three of them onto an Allegro hand is a mechanical engineering project; fitting five onto a Shadow hand or a humanoid hand is a research paper on its own. The mismatch means that even when tactile data is collected on one platform, retargeting it to the deployment platform requires assumptions the community has not yet earned.

The third reason is that vision-tactile signals are hard to compress into the same tensor shape as the rest of the observation stream. A camera frame slots into a policy naturally. A DIGIT frame nominally does too, but its statistics are so different from a scene camera, texture-dominated, near-uniform lighting, sub-pixel detail carrying most of the information, that naive concatenation underperforms. Serious use requires a separate encoder, which requires a separate training signal, which requires enough tactile data to train the encoder in the first place. The chicken and egg is real.

The specific capabilities that vision-tactile enables, slip prediction, contact-patch geometry, shear-aware grasp control, are exactly the capabilities that separate a robot that can pick up a wrench from a robot that can tighten a bolt. Every foundation-model policy trained without those signals is inheriting a ceiling. It can learn to reach and grasp. It cannot learn the last few grams of a task.

Reviewers notice this and it shapes what gets published. The set of tasks reported in flagship foundation-model papers has drifted toward the tasks the sensor mix can support: pick, place, wipe, push, open. Contact-rich, sub-millimeter, force-critical tasks, the industrial long tail, appear as gestures rather than as benchmarks.

The response the field is quietly taking is to shift the tactile burden off the robot fingertip and onto the human demonstrator. If a vision-tactile sensor is too fragile and too large to sit on every robot at scale, instrument the human instead. Put force sensors on the fingers of the person doing the demonstration, capture at rates the human corrections actually contain, and let the policy learn from a labeled human record that already includes the contact signals.

This is not a rejection of GelSight-class sensors. It is a division of labor. Vision-tactile stays where it belongs, on the robot fingertip at deployment time, providing a live signal to the running policy. Wearable force capture takes the training-data role, because that is the role where its scaling properties are actually workable.

The next piece describes how we made that trade concretely at BLO LAB and why the resulting stack looks the way it does.