Sim-to-Real Transfer: Where It Breaks and How Real Data Fixes It

Simulation is the cheapest way to produce a lot of training data. Millions of episodes for the cost of some GPUs, run overnight, no operators, no lunch breaks, no calibration drift. Naturally, teams try to lean on it. Naturally, reality bites. This article maps where the reality gap opens and what actually closes it.

Sim renders are not photos. Even the best physically-based renderers miss subsurface scattering, sensor noise characteristics, motion blur under real exposure, and the long tail of optical artifacts a real camera exhibits. A policy trained on sim vision overfits to the simulator's rendering pipeline, not the world.

Sim contact models are not physics. Rigid-body physics engines approximate friction, restitution, and deformation with tunable parameters that are 'close enough' for most rigid interactions and 'nowhere near close enough' for cloth, food, or granular materials. A sim policy for kneading dough is a policy for a fictional substance.

Sim latency is not the real control loop. Real robots have communication jitter, actuator delays, and sensor latency that vary from cycle to cycle. Sim runs deterministically. A policy that only ever saw perfect timing will misfire the moment it meets an imperfect one.

The standard mitigation is to randomize everything randomizable during simulation: lighting, textures, friction coefficients, object masses, sensor noise levels. The theory is that a policy trained across a wide enough distribution will generalize to the narrower distribution of the real world.

In practice it caps out. The randomization distribution is a guess, and the guess is usually wrong at the tails. If the real friction coefficient sits outside the randomized range, the policy fails. If real motion blur has a different profile than the noise you injected, the vision encoder mismatches. Randomization buys you an order of magnitude of robustness. It does not buy you two.

A modest set of real teleop or mocap trajectories, mixed into the training set alongside sim data, dramatically raises real-world performance. This co-training pattern is now standard practice at every serious lab. Ratios vary, 10:1 sim-to-real is common, 3:1 is used when the real data is very high quality, but the direction is universal.

The reason it works is that real data anchors the policy in the actual sensor distribution and the actual contact dynamics. Sim data provides the coverage and variety that would be prohibitively expensive to capture. The two are complementary, not competitive.

Locomotion. Legged robots trained purely in sim with heavy randomization now walk, run, and recover on hardware they have never touched. The reason: locomotion is a proprioceptive control task with well-modeled rigid-body dynamics. Manipulation, especially contact-rich manipulation, is neither.

Sim is also excellent for the perception frontend: pretraining depth prediction, segmentation, and object pose estimation on synthetic scenes and then fine-tuning on a small real set. The perception model transfers; the policy model needs the real trajectories.

Use sim for what it is good at: massive coverage of scene variation, pretraining of perception, and locomotion. Use real capture for what sim cannot deliver: contact dynamics, force, human-like trajectory shape, and the fine-grained sensor distribution the deployed robot will actually see. Co-train, do not choose.