Towards Fast Safe Online Reinforcement Learning via Policy Finetuning

Keru Chen; Honghao Wei; Zhigang Deng; Sen Lin

Marvel Offline-to-online safe reinforcement learning

Towards Fast Safe Online Reinforcement Learning via Policy Finetuning

Keru Chen, Honghao Wei, Zhigang Deng, Sen Lin

Transactions on Machine Learning Research · January 2026

Read paper PDF arXiv TMLR / OpenReview arXiv DOI Code BibTeX

Abstract

High costs and risks involved in extensive environmental interactions hinder the practical application of current online safe reinforcement learning (RL) methods. Inspired by recent successes in offline-to-online (O2O) RL, it is crucial to explore whether offline safe RL can be leveraged to facilitate faster and safer online learning, a direction that has yet to be fully investigated. To fill this gap, we first show that naively applying existing O2O algorithms from standard RL would not work well in safe RL due to two unique challenges: erroneous Q-estimations, resulted from offline-online objective mismatch and offline cost sparsity, and Lagrangian mismatch, resulted from difficulties in aligning Lagrange multipliers between offline and online policies. To address these challenges, we introduce Marvel, the first policy-finetuning based framework for O2O safe RL, comprising two key components that work in concert: Value Pre-Alignment to align the learned Q-functions with the online objective before finetuning, and Adaptive PID Control to effectively adjust the Lagrange multipliers during finetuning. Extensive experiments demonstrate the superior performance of Marvel over related baselines.

Original abstract · Published TMLR version

Problem statement

A safe policy learned from offline data is a useful starting point, but its value estimates were trained to discourage actions outside that dataset. Once online interaction becomes available, those conservative estimates can hold back improvement even when the policy operates far below its cost limit. Sparse cost observations make learning the cost value function difficult as well.

The Lagrange multiplier introduces a second mismatch. It controls how strongly online learning penalizes cost, but a suitable initial multiplier is not automatically available from the pretrained policy. Simply transferring the policy and its critics can therefore produce either stagnant learning or unsafe updates.

BallCircle reward and cost curves for Warm Start, From Scratch, and Marvel, alongside t-SNE views showing sparse cost observations. — Naive warm starts improve slowly on BallCircle. The right panels illustrate sparse cost observations. Steps are optimizer updates, each involving three episodes; the dashed line marks the cost limit of 20. Figure 1

Method

Marvel first performs Value Pre-Alignment (VPA). It keeps the pretrained policy fixed and updates the reward and cost Q-functions on offline data to match the online objective. This prepares the critics for finetuning without requiring new environment interactions for the alignment step.

During online finetuning, Adaptive PID Control (aPID) adjusts the Lagrange multiplier in response to the observed cost and the constraint threshold. The two components address different parts of the transition: VPA changes the initial value estimates, while aPID controls the penalty as learning proceeds.

Key findings

The comparisons below use a cost limit of 20 and five random seeds. Unless stated otherwise, CPQ supplies the offline policy and SAC-lag performs online finetuning. An update means a policy-gradient optimizer update, which involves three episodes; it is not one environment timestep.

1. Finetuning improves earlier than a naive warm start

On BallCircle after 100 updates, Marvel reaches 580.3 reward / 19.2 cost. Warm Start remains at 54.7 / 0.0, while learning From Scratch gives 170.8 / 88.9. The latter exceeds the cost limit. At 500 updates, Marvel reaches 603.9 / 19.8. Table 2, paper p. 9

BallCircle, 100 updates	Reward	Cost (limit 20)
Warm Start	54.7	0.0
From Scratch	170.8	88.9
Marvel	580.3	19.2

2. The gain also appears on CarCircle

At 500 updates, Marvel obtains 341.3 reward / 19.5 cost, compared with SO2’s 115.9 / 9.1 and From Scratch’s 115.2 / 12.7. Warm Start reaches 141.6 / 24.1, exceeding the cost threshold. Table 2, paper p. 9

3. Pre-alignment improves the ranking of value estimates

For BallCircle state-action pairs sampled from the offline dataset, VPA changes the Spearman correlation with Monte Carlo estimates from −0.3852 to 0.8278 for reward Q-values and from 0.1725 to 0.8252 for cost Q-values. These are ranking correlations, not absolute estimation errors. Table 1, paper p. 6

4. Both pretrained critics contribute to the result

On BallCircle, full VPA yields 603.9 reward / 19.8 cost. Updating only reward Q without entropy gives 364.7 / 11.2, while updating only cost Q without entropy gives 64.1 / 5.3. Reinitializing both critics gives 1.2 / 51.0. The ablation supports refining both pretrained value functions in this setting. Table 4, paper p. 12

5. The framework also supports a different offline initializer

With BEAR-lag pretraining on BallCircle, Marvel reaches 552.3 reward / 15.2 cost at 500 updates, compared with Warm Start’s 440.2 / 100.0. This extends the evidence beyond CPQ, without establishing compatibility with every offline algorithm. Table 2, paper p. 9

Relation to prior work

Warm Start directly transfers the policy and critics. Marvel instead aligns the critics with the online objective before making policy updates. SO2 improves Q-estimation through perturbed value updates; Marvel explicitly addresses the offline–online objective mismatch and sparse cost evidence in a separate alignment phase.

JSRL and PEX use pretrained policies to guide exploration. Marvel finetunes the policy and critics themselves. The comparison baselines in this paper also receive aPID, so the reported gains should not be attributed simply to giving only Marvel an adaptive controller. Experimental setup, Section 4.1

Limitations

Marvel does not guarantee safety throughout learning. The paper reports early cost spikes; for example, CarRun with BEAR-lag has cost 60.0 after 100 updates, above the limit of 20. AntRun is a failure setting where reward and cost do not improve substantially during finetuning.

The VPA analysis depends on data coverage and approximation assumptions. Value estimates for out-of-distribution actions can remain optimistic when coverage is imperfect. Experiments cover ten simulated environments in Bullet Safety Gym and Safety Gymnasium, rather than hardware deployment. Adapting the derivative gain to new nonstationary phases remains future work. Discussion and limitations, Appendix I

Resources

The DOI resource above identifies the arXiv version. The BibTeX below cites the published TMLR article through OpenReview.

BibTeX

Download .bib

@article{chen2026towards,
  title = {Towards Fast Safe Online Reinforcement Learning via Policy Finetuning},
  author = {Chen, Keru and Wei, Honghao and Deng, Zhigang and Lin, Sen},
  journal = {Transactions on Machine Learning Research},
  year = {2026},
  month = jan,
  url = {https://openreview.net/forum?id=1SO7vmLFUq}
}