WEBVTT

NOTE
English narration from the final v10 approved script. Scene timing follows
the final audio assembly; sentence boundaries are proportionally estimated.

1
00:00:00.200 --> 00:00:05.596
Race: Calibrated Cumulative Scoring for Rollout-Free
Checkpoint Selection.

2
00:00:06.446 --> 00:00:09.795
During training, we save checkpoints at regular
intervals.

3
00:00:09.795 --> 00:00:13.982
But lower training loss does not guarantee higher task
success.

4
00:00:14.832 --> 00:00:19.256
Measuring every checkpoint requires repeated robot
trials and resets.

5
00:00:19.256 --> 00:00:27.120
A common alternative is averaged prediction error: the
average mismatch between predicted actions and held-out
demonstrations.

6
00:00:27.120 --> 00:00:29.578
But this can mislead selection.

7
00:00:30.428 --> 00:00:32.109
Consider two policies.

8
00:00:32.109 --> 00:00:34.910
A imitates closely but fails.

9
00:00:34.910 --> 00:00:37.712
B deviates more but succeeds.

10
00:00:37.712 --> 00:00:41.634
Yet lower mean squared error favors A.

11
00:00:41.634 --> 00:00:42.194
Why?

12
00:00:43.494 --> 00:00:47.439
Smaller recoverable deviations need not improve success.

13
00:00:47.439 --> 00:00:50.820
Averaging can hide A's fatal mistake.

14
00:00:51.670 --> 00:00:58.286
Race therefore gives equal scores to deviations within a
threshold, and penalizes deviations beyond it.

15
00:00:59.136 --> 00:01:07.446
For each action group, Race sets a high-quantile
threshold from action differences at similar states in
successful references.

16
00:01:07.446 --> 00:01:11.139
We combine group scores into a timestep score.

17
00:01:11.989 --> 00:01:19.065
Averaging also counts later predictions, even when
earlier deviations could take the robot away from those
reference states.

18
00:01:19.915 --> 00:01:26.254
Race takes a running product of timestep scores,
reducing later contributions after a low score.

19
00:01:26.254 --> 00:01:29.212
Averaging this curve penalizes earlier departures more.

20
00:01:30.062 --> 00:01:33.998
We average across references and rank checkpoints:
higher is better.

21
00:01:34.848 --> 00:01:40.604
We test eight DexMimicGen tasks, using three sources of
successful reference trajectories.

22
00:01:41.454 --> 00:01:43.700
Ten held-out demonstrations.

23
00:01:44.250 --> 00:01:47.536
Ten successful policy rollouts.

24
00:01:48.086 --> 00:01:51.888
Or ten successful rollouts collected with action noise.

25
00:01:52.738 --> 00:01:58.760
Mean maximum rank violation measures ranking errors,
penalizing larger success-rate gaps more heavily.

26
00:01:58.760 --> 00:02:00.150
Lower is better.

27
00:02:00.150 --> 00:02:08.024
Race maintains low error across all three reference
sources, with the lowest mean among the compared
methods.

28
00:02:08.874 --> 00:02:15.120
Next, we evaluate a vision-language-action policy on
three real-world manipulation tasks.

29
00:02:15.970 --> 00:02:18.204
Each point represents a checkpoint.

30
00:02:18.204 --> 00:02:24.906
Race scores track measured success more closely than the
compared baselines, with lower ranking error.

31
00:02:25.756 --> 00:02:29.000
We also evaluate DROID diffusion policies.

32
00:02:29.850 --> 00:02:36.910
The mismatch is even stronger: lower validation loss or
MSE can favor worse checkpoints.

33
00:02:36.910 --> 00:02:39.936
Race remains aligned with measured success.

34
00:02:40.786 --> 00:02:46.601
Finally, compare Race scores with checkpoint success
rates measured through repeated robot trials.

35
00:02:46.601 --> 00:02:50.180
Across checkpoints, the two curves follow similar
trends.

36
00:02:50.180 --> 00:02:58.232
Race captures these performance differences using
held-out demonstrations collected in advance, without
running each candidate on the robot.
