DEAS: DEtached value learning with Action Sequence for Scalable Offline RL

Changyeon Kim¹ Haeone Lee¹ Younggyo Seo² Kimin Lee^{^1} Yuke Zhu^{^3,4}

¹ KAIST ² UC Berkeley ³ The University of Texas at Austin ⁴ NVIDIA

ICLR 2026

Paper Code (RoboCasa) Dataset (RoboCasa) Code (OGBench)

TL;DR & Key Insights

DEAS is a simple yet effective offline RL framework that learns a critic over H-step action sequences with detached value learning, enabling stable, plug-and-play RL enhancement of VLA models.

Detached value learning. Decouple the critic from the actor to avoid value overestimation over expanded action spaces.
Distributional RL + dual discount factors. Categorical values with separate intra- / inter-sequence discounts γ₁, γ₂^H stabilize multi-step returns.
Plug-and-play with any policy, including large VLAs. Boosts GR00T N1.5 / pi_0 on RoboCasa Kitchen and real-world Franka manipulation without architectural changes.

Overview

Previous Methods: Value Overestimation over Expanded Action Spaces

Prior actor-critic methods that adopt action sequences let the policy π propose a full H-step sequence â_H:2H−1. Because this expanded action space is far larger than what the offline dataset supports, the target critic Q̄ is queried on out-of-distribution sequences, and the actor is free to exploit the resulting critic errors — causing severe value overestimation and unstable training in offline settings.

Illustration of value overestimation in previous action-sequence actor-critic methods

DEAS: Detached Value Learning with Action Sequences

DEAS removes the actor from the critic update loop entirely. A value network V(s) and action-sequence critic Q(s, a_0:H−1) are trained with IQL-style expectile regression (the τ·-weighted target), biasing value estimates toward high-return, in-distribution action sequences. We further combine this with distributional RL — categorical distributions over a fixed value support — and dual discount factors γ₁, γ₂ that separately control reward aggregation within a sequence and bootstrapping across sequence-level decision points. The policy is then extracted with any off-the-shelf method (BoN, DPG, AWR, etc.), which makes DEAS directly applicable to expressive policies including large-scale VLAs.

Experiments

Franka Research 3 Kitchen

We report the partial success rate (%, over 20 trials per task) on 3 tasks from 5 initial points. Bold and underline indicate best and runner-up results, respectively.

DEAS demonstrates consistent performance improvements across all tasks In contrast, baseline methods show inconsistent performance—while some methods perform well on certain tasks, they exhibit significant performance degradation or minimal improvement on others.

Peach

All videos are 2x real-time.

GR00T N1.5

Filtered BC

IQL

DEAS (Ours)

Hichew

All videos are 2x real-time.

GR00T N1.5

Filtered BC

IQL

DEAS (Ours)

RoboCasa Kitchen

We report the success rate (%, over 50 trials per task) on 4 tasks, aggregated with 3 different seeds. Bold and underline indicate best and runner-up results, respectively.

DEAS achieves the highest success rates in 3 out of 4 tasks, with the remaining task also showing improved performance compared to the base model.

CoffeeSetupMug

GR00T N1.5

Filtered BC

IQL

DEAS (Ours)

PnPCounterToMicrowave

GR00T N1.5

Filtered BC

IQL

DEAS (Ours)

PnPMicrowaveToCounter

GR00T N1.5

Filtered BC

IQL

DEAS (Ours)

TurnOffStove

GR00T N1.5

Filtered BC

IQL

DEAS (Ours)

OGBench

DEAS consistently outperforms various prior offline RL methods in OGBench, and shows far more effective performance in more challenging tasks (e.g., puzzle and cube-quadruple). Furthermore, DEAS shows consistent performance across different data scales in diverse tasks.

Citation

@inproceedings{kim2026deas,
    title={DEAS: DEtached value learning with Action Sequence for Scalable Offline RL},
    author={Changyeon Kim and Haeone Lee and Younggyo Seo and Kimin Lee and Yuke Zhu},
    booktitle={International Conference on Learning Representations (ICLR)},
    year={2026},
}