Evaluating Robot Policies: Metrics That Actually Matter

Every robotics paper reports success rate. Success rate hides more than it reveals. Two policies with the same success rate can behave completely differently on the metrics that actually matter for deployment. Better evaluation uses a small basket of metrics that together paint an honest picture.

Success rate is a binary aggregation over a hidden distribution. A 90% success rate can be 100 trials at 90% or ten trials at 90% or a hundred trials of which the ninety successes were the easy ones and the ten failures were the hard ones. Reporting it without confidence intervals and without a breakdown by difficulty is close to reporting nothing.

This is not an argument against success rate; it is an argument against reporting only success rate.

How long from the observation-conditioned start of the episode to the moment the robot first touches the target object? This is a proxy for perception latency and for policy decisiveness. A policy that stares before acting is a policy that will lag in production and will feel slow to users.

Given a partial failure, a grasp that slips, an object that gets knocked over, a tool that misses the target, how often does the policy self-correct and complete the task? This separates brittle policies from robust ones more sharply than success rate does. A policy that never fails on easy trials but never recovers on hard ones is a policy that will not survive the long tail.

The time-derivative of applied force during contact-rich phases. Jerky force profiles crack eggs, dent packaging, and scare humans. Smooth force profiles are a proxy for policy maturity and a strong predictor of how a policy will feel to users in a shared space.

In semi-supervised deployment, how often does a human have to take over? This is the most honest single metric for practical readiness. A policy with a 95% success rate and a 30% intervention rate is not a shipping product. A policy with a 90% success rate and a 5% intervention rate might be.

Aggregate metrics hide the tail. Report success rate per object, per environment, per operator style. The variance across these slices is often larger than the variance between competing policies, which means it is often more informative.

Design your evaluation harness before you train your first policy. Log everything, aggregate along multiple axes, and publish confidence intervals. A team that evaluates rigorously will iterate faster than a team that trains more, because the second team will chase phantom improvements the first team can see through.