RL-Pretrained Action Experts for VLAs

Towards physical priors in diffusion-policy initialization

Research proposal · ZenodoSolo-authored position paper

Vision-Language-ActionDiffusion PolicyPPO / DPPO / DDiffPGMuJoCo · Isaac Lab

1Simulator RL

Train a control policy in a physics simulator (MuJoCo, Isaac Lab) with reinforcement learning — PPO, DPPO, or DDiffPG — under a generic physics reward. Task-agnostic and cheap to run.

2Pretrained trunk θ_AE

The result is a small causal transformer or diffusion policy whose weights already encode how to move — walking and whole-body skills baked in.

3Convert to a DiT

Reshape that trunk into a diffusion action expert: add adaLN-Zero, turn some self-attention into cross-attention, and swap in embodiment I/O MLPs. (Direct transfer for Paradigm 2.)

4Attach + fine-tune

Bolt it onto a frozen VLM, let it cross-attend the VLM's tokens, and only now fine-tune on demonstrations — for any task, any embodiment.

RL before SFT

A VLA whose action expert already understands physics

locomotion · whole-body control · disturbance recovery — before it sees a single demonstration

TL;DR

Modern Vision-Language-Action models bolt a randomly-initialized diffusion action expert onto a pretrained VLM and teach it to move purely from demonstrations. Meanwhile, reinforcement learning in physics simulators already produces controllers that walk over rough terrain and perform whole-body acrobatics — from the same transformer family as that action expert. This proposal bridges the two: use simulator RL to pretrain the action expert against a generic physics reward, installing a task-agnostic physical prior before any demonstration is seen. I show it is feasible with two established architectures and lay out exactly how each converts into a VLA action expert.

RL → SFT

Reverse the usual order: RL before fine-tuning

2 paradigms

PPO causal transformer · DPPO / DDiffPG diffusion policy

0% → 95%

The kind of gain RL pretraining bought on LIBERO-Spatial^[10]

Two worlds that don't talk

Vision-Language-Action (VLA) models are the leading candidate for a generalist robot policy. By attaching an action decoder to a pretrained Vision-Language Model (VLM), they inherit internet-scale semantics and expose a robot through a natural-language interface — the recipe shared by the Physical Intelligence $\pi$ family^[1], NVIDIA GR00T^[2], SmolVLA^[3] and X-VLA^[4]. But these models are trained by supervised fine-tuning (SFT) on demonstrations, and their action decoders start from random weights.

In parallel, model-free RL in modern simulators — MuJoCo^[5] and Isaac Lab^[6] — has produced a second generation of controllers with capabilities no supervised pipeline has matched: humanoids that walk miles over challenging terrain^[7] and the viral clips of robots dancing or doing martial arts are, almost without exception, the product of large-scale simulator RL from an unmodified per-step reward. These controllers, though, have no language and no vision interface.

The two fields barely touch. NVIDIA's SONIC^[9] partially bridges them by training a separate whole-body motion-tracking policy on 700 hours of motion capture and routing commands to it through a fine-tuned GR00T VLA — a System-1 / System-2 split that needs two trained models, VR teleoperation hardware, and a large mocap corpus. This proposal takes the other road: rather than maintain the split, instill a physical sense directly into the VLA's own action-expert weights via simulator RL, so a single model is trained end-to-end and demonstrations are the only downstream data required.

The walking gap. Today's VLAs walk poorly, slowly, or only via a separately-trained low-level controller — while a 1.4M-parameter RL-trained transformer walks a Digit humanoid zero-shot over rough terrain^[7]. The VLA action expert and the RL transformer are close architectural cousins; the community has built every piece of the bridge — the simulators, the architectures, the algorithms — and simply not crossed it in this direction.

Background — the three pieces

A modern VLA decomposes into three modules: (1) a pretrained VLM backbone that consumes images and language and emits token embeddings; (2) an action expert, architecturally a Diffusion Transformer (DiT)^[16], that consumes noised action tokens plus the proprioceptive state, with the VLM tokens routed in through cross-attention; and (3) embodiment-specific projection MLPs that map the robot's state and action vectors to and from a shared latent dimension. VLAs are trained by behavior cloning on expert trajectories, minimizing a supervised loss over predicted action chunks:

\mathcal{L}_{BC}(\theta) = \mathbb{E}_{(o_n, A_n)\sim\mathcal{D}}\big[\,\ell\big(\pi_\theta(o_n),\, A_n\big)\big]

(1)

An RL control policy, by contrast, is trained in a parallelized GPU simulator to maximize a hand-designed reward. On-policy methods such as PPO^[11] dominate simulator-scale training because they are stable and parallelize well; off-policy methods learn a value function from a replay buffer and are more sample-efficient but harder to stabilize at scale. A Diffusion Policy^[12] parameterizes the action distribution as the reverse of a noising process — flow matching^[13] is the continuous-time special case used by modern VLA action experts. Two recent works lift diffusion policies to RL: DPPO^[14] treats the denoising chain as an inner MDP that PPO can optimize, and DDiffPG^[15] trains a diffusion policy from scratch with off-policy actor-critic. The takeaway: a diffusion policy with the same architecture as a VLA action expert can be trained from environment reward alone.

The initialization problem

Treat VLA control as a POMDP. The policy outputs an action chunk of horizon $H$ given an observation history of length $T$ and a language instruction $\ell$ :

\pi_\theta\big(a_{t:t+H-1} \,\mid\, o_{t-T+1:t},\ \ell\big)

Partition the parameters into backbone, action-expert trunk, and embodiment MLPs, $\theta = (\theta_{\mathrm{VLM}},\, \theta_{\mathrm{AE}},\, \theta_{\mathrm{embod}})$ . Standard practice initializes the backbone from a web-scale checkpoint, $\theta_{\mathrm{VLM}}^{(0)} = \theta_{\mathrm{VLM}}^{\mathrm{web}}$ , and draws $\theta_{\mathrm{AE}}^{(0)}$ and $\theta_{\mathrm{embod}}^{(0)}$ from a random initializer; SFT then minimizes a flow-matching loss $\mathcal{L}_{\mathrm{fm}}(\theta;\, \mathcal{D}_{\mathrm{demo}})$ .

The bottleneck is that $\theta_{\mathrm{AE}}^{(0)}$ encodes no inductive bias about actuation. The flow-matching loss must therefore use demonstrations to teach two things at once:

how to denoise toward a clean action distribution; and
the basic dynamics of the embodiment.

Demonstrations are scarce, expensive, and biased toward expert behavior — recovery from disturbances and the resulting robustness are weakly represented in them. RL in simulation has, for nearly a decade, been the substrate that solves that second point. This is the gap the proposal fills: pretrain $\theta_{\mathrm{AE}}$ with RL so SFT only has to specialize a competent controller, not learn physics from scratch.

The proposal — RL before SFT

I identify two pretraining paradigms, each grounded in established work and each producing a trunk that transfers into a DiT-style action expert.

Paradigm 1

Causal transformer via standard PPO

A small causal transformer in the same family as the target action-expert trunk, taking a context window of past proprioceptive observations and actions — as in the Berkeley humanoid controllers (HT-2: 4 blocks, hidden dim 192, 4 heads, context 16, ~1.4M params). Trained by PPO on an unmodified per-step reward.

Paradigm 2

Diffusion policy via DPPO or DDiffPG

A transformer-based diffusion / flow-matching policy — the VLA action expert minus the cross-attention to VLM tokens. Trained on-policy with DPPO, or from scratch off-policy with DDiffPG. Weight transfer at the end of pretraining is direct.

Paradigm 1 · the PPO objective

PPO optimizes the clipped surrogate, where $\hat{A}_t$ is an advantage estimate from the unmodified per-step reward and $\epsilon$ bounds the per-update policy change:

\mathcal{L}^{\mathrm{PPO}}(\theta) = \mathbb{E}_t\!\Big[\min\!\big(r_t(\theta)\,\hat{A}_t,\ \operatorname{clip}(r_t(\theta),\, 1-\epsilon,\, 1+\epsilon)\,\hat{A}_t\big)\Big], \qquad r_t(\theta) = \frac{\pi_\theta\big(a_t \mid o_{t-T+1:t}\big)}{\pi_{\theta_{\mathrm{old}}}\big(a_t \mid o_{t-T+1:t}\big)}

(2)

Converting that causal transformer into a DiT-style action expert is three mechanical operations:

Insert adaLN. Wrap each block in adaptive layer norm whose scale and shift are regressed from the flow-matching timestep embedding, with the residual scale initialized to zero (adaLN-Zero^[16]) so every block is the identity at the start of SFT and the RL-pretrained forward function is preserved.
Replace the input/output projections with the embodiment-specific state-and-action embedder MLP, sized to the new state and action dimensions — the cross-embodiment pattern of GR00T^[2].
Convert some self-attentions to cross-attentions onto the frozen VLM's token embeddings, mirroring the encoder-to-decoder cross-attention of the original Transformer^[17]. The VLM itself is untouched.

Paradigm 2 · the diffusion-policy MDP

DPPO^[14] unrolls the denoising process into an inner MDP nested inside the environment MDP. In the composite “Diffusion Policy MDP”, reward flows only at the fully-denoised step $k=0$ :

\bar{R}_{\bar{t}(t,k)}(\bar{s},\bar{a}) = \begin{cases} 0, & k > 0,\\[4pt] R\big(s_t,\, a_t^{0}\big), & k = 0, \end{cases}

(3)

and the Gaussian likelihood of each denoising step admits exact policy-gradient updates:

\pi_\theta\big(a_t^{k} \mid a_t^{k+1}, s_t\big) = \mathcal{N}\!\big(a_t^{k};\ \mu\big(a_t^{k+1},\, \varepsilon_\theta(a_t^{k+1}, k+1, s_t)\big),\ \sigma_k^{2} I\big)

In practice one fine-tunes only the last few denoising steps, runs inference with DDIM, and clamps a minimum exploration noise. DDiffPG^[15] is the alternative when no demonstrations exist for the pretraining task: it trains the diffusion policy from scratch with off-policy actor-critic, a Random Network Distillation novelty bonus, and a per-mode Q-function.

Transfer to the VLA action expert

Both paradigms produce a pretrained trunk $\theta_{\mathrm{AE}}$ that initializes the action expert. The full assembly loads the VLM backbone from its public checkpoint, initializes $\theta_{\mathrm{AE}}$ from the simulator-trained checkpoint (with the architectural conversion above for Paradigm 1, or direct weight transfer for Paradigm 2), and randomly initializes the embodiment MLPs $\theta_{\mathrm{embod}}$ ; the VLM is typically frozen. For SFT itself, two regimes are worth comparing empirically — adapter-first (freeze the trunk, warm up only the new layers) versus joint (unfreeze everything from step one). Neither is a priori better; the choice is left to ablation.

How it differs from prior work

Not RL fine-tuning of VLAs. A growing line applies RL after SFT to refine task success — iRe-VLA^[18], ConRFT^[19], SimpleVLA-RL^[20], $\pi_{\mathrm{RL}}$ ^[21] and VLA-RFT^[22] all share an SFT-first, RL-second template. This proposal is the opposite ordering: RL before SFT, on a different (simulated) domain. The two are complementary.
Not task-specific RL pretraining. The closest prior work, ActionX^[10], also RL-pretrains a flow-matching action expert with a frozen VLM before SFT, and shows the ordering beats both supervised pretraining and no pretraining (LIBERO-Spatial success rising from 0% to 95%). But ActionX's prior is task-specific, manipulation-only, and single-embodiment. The prior proposed here is trained once against a generic physics reward, covers whole-body skills such as locomotion, and is meant to transfer across many tasks and different embodiments.
Not offline RL pretraining. RECAP / $\pi^{*}_{0.6}$ ^[23] uses offline RL to train a value-function VLM that emits an advantage label; the action-expert weights stay random and are still trained by behavior cloning. Here the RL signal is directed at the action-expert weights themselves, inexpensively, inside a physics simulator.

Limitations & open questions

Action-space mismatch. The upstream controller may output joint setpoints with PD gains at a fixed control frequency, while a downstream VLA produces end-effector deltas or base velocities at chunk frequency. The embodiment MLPs absorb this in principle, but the upstream run must be designed with the downstream action representation in mind.
Scale. VLA action experts approach a billion parameters (300M in $\pi_0$ , 860M in $\pi_{0.6}$ ) while the established RL-trained upstream policies are orders of magnitude smaller (HT-2 at 1.4M, SONIC up to ~42M). Either the transferred weights occupy a small fraction of the action expert, or the RL run has to be scaled up to match.
It is a proposal. Every architectural and algorithmic component already exists; what remains is the empirical study — of the RL-before-SFT ordering, and of how broadly the resulting physics prior transfers across embodiments and tasks.

This write-up summarizes a position paper I posted on Zenodo. It proposes the recipe and grounds every piece in existing architectures and algorithms; the empirical validation is deliberately left as follow-up work. The full PDF is here.

References

[1] K. Black, N. Brown, D. Driess, et al. (2026). π₀: A vision-language-action flow model for general robot control. arXiv:2410.24164.
[2] NVIDIA, J. Bjorck, F. Castañeda, et al. (2025). GR00T N1: An open foundation model for generalist humanoid robots. arXiv:2503.14734.
[3] M. Shukor, D. Aubakirova, F. Capuano, et al. (2025). SmolVLA: A vision-language-action model for affordable and efficient robotics. arXiv:2506.01844.
[4] J. Zheng, J. Li, Z. Wang, et al. (2025). X-VLA: Soft-prompted transformer as scalable cross-embodiment vision-language-action model. arXiv:2510.10274.
[5] E. Todorov, T. Erez, Y. Tassa (2012). MuJoCo: A physics engine for model-based control. IEEE/RSJ IROS, 5026–5033.
[6] M. Mittal, P. Roth, J. Tigue, et al. (2025). Isaac Lab: A GPU-accelerated simulation framework for multi-modal robot learning. arXiv:2511.04831.
[7] I. Radosavovic, S. Kamat, T. Darrell, J. Malik (2024). Learning humanoid locomotion over challenging terrain. arXiv:2410.03654.
[8] I. Radosavovic, T. Xiao, B. Zhang, T. Darrell, J. Malik, K. Sreenath (2023). Real-world humanoid locomotion with reinforcement learning. arXiv:2303.03381.
[9] Z. Luo, Y. Yuan, T. Wang, et al. (2025). SONIC: Supersizing motion tracking for natural humanoid whole-body control. arXiv:2511.07820.
[10] T. Yi, Q. Yang, E. Chen (2026). ActionX: pre-training action experts with reinforcement learning for vision-language action models. Frontiers in Neurorobotics 20:1806605.
[11] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, O. Klimov (2017). Proximal policy optimization algorithms. arXiv:1707.06347.
[12] C. Chi, Z. Xu, S. Feng, et al. (2024). Diffusion policy: Visuomotor policy learning via action diffusion. arXiv:2303.04137.
[13] Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, M. Le (2023). Flow matching for generative modeling. arXiv:2210.02747.
[14] A. Z. Ren, J. Lidard, L. L. Ankile, et al. (2024). Diffusion policy policy optimization (DPPO). arXiv:2409.00588.
[15] Z. Li, R. Krohn, T. Chen, A. Ajay, P. Agrawal, G. Chalvatzaki (2024). Learning multimodal behaviors from scratch with diffusion policy gradient (DDiffPG). arXiv:2406.00681.
[16] W. Peebles, S. Xie (2023). Scalable diffusion models with transformers (DiT). arXiv:2212.09748.
[17] A. Vaswani, N. Shazeer, N. Parmar, et al. (2017). Attention is all you need. arXiv:1706.03762.
[18] Y. Guo, J. Zhang, X. Chen, X. Ji, Y.-J. Wang, Y. Hu, J. Chen (2025). Improving vision-language-action model with online reinforcement learning (iRe-VLA). arXiv:2501.16664.
[19] Y. Chen, S. Tian, S. Liu, Y. Zhou, H. Li, D. Zhao (2025). ConRFT: A reinforced fine-tuning method for VLA models via consistency policy. arXiv:2502.05450.
[20] H. Li, Y. Zuo, J. Yu, et al. (2025). SimpleVLA-RL: Scaling VLA training via reinforcement learning. arXiv:2509.09674.
[21] K. Chen, Z. Liu, T. Zhang, et al. (2025). π_RL: Online RL fine-tuning for flow-based vision-language-action models. arXiv:2510.25889.
[22] H. Li, P. Ding, R. Suo, et al. (2025). VLA-RFT: Vision-language-action reinforcement fine-tuning with verified rewards in world simulators. arXiv:2510.00406.
[23] Physical Intelligence, K. Black, A. Amin, et al. (2025). π*₀.₆: a VLA that learns from experience (RECAP). arXiv:2511.14759.