Information-Directed Offline-to-Online Reinforcement Learning

Keru Chen

doi:10.48550/arXiv.2605.29405

IDS Exploration after offline learning

Information-Directed Offline-to-Online Reinforcement Learning

Keru Chen

NeurIPS 2026 · Main Conference · Also at ICML 2026 DEMO Workshop

Read paper PDF arXiv NeurIPS arXiv DOI BibTeX

Abstract

Decision-making from offline datasets typically warm-starts a policy or score model from fixed offline data and then refines it with limited online interaction. Offline data reduces uncertainty, but it does not remove the need for exploration; it changes what remains to be explored. We formalise this residual uncertainty by the conditional mutual information $I(\chi;\tau_{1:T}\mid\mathcal{D}_{N})$ between a learning target $\chi$ and the online trajectories after conditioning on the offline dataset. This view leads naturally to information-directed sampling (IDS), a family parameterised by $\eta\geq 0$ that selects actions by trading off instantaneous regret against information gain. We prove a generic offline-to-online Bayesian regret bound for IDS through a ratio certificate: any information-ratio bound satisfied by a reference Thompson-sampling policy over the same randomised policy class is inherited by IDS. In a known-dynamics Bayesian linear-reward model, the conditional mutual information has a log-determinant form, and vanilla IDS ( $\eta=0$ ) satisfies $\widetilde{O}\!\left(Hd\min\left\{\sqrt{T},\,T\sqrt{C^{\dagger}_{\beta,\mathrm{IDS}_{0}}(N,T)/N}\right\}\right),$ where the coverage coefficient is tied to the visitation distribution induced by vanilla IDS itself. We also identify a warm-start regime with a dominated but informative probe in which vanilla IDS selects the probe while Thompson sampling never does, giving a constant-factor Bayesian regret separation. Controlled bandit experiments and D4RL offline-to-online RL experiments validate this mechanism: IDS is most beneficial when offline data is informative but leaves biased or low-probability residual uncertainty that targeted online actions can resolve, a regime shared by offline RL, offline black-box optimization, and Bayesian optimization.

Original abstract · arXiv:2605.29405v1 · May 28, 2026

Problem statement

An offline dataset can make a policy confident without resolving every uncertainty that matters. Most of the environment may already be understood, while a rare alternative remains plausible. Exploring broadly wastes the information supplied by the dataset; following the current best policy may never test the remaining alternative.

The question is therefore specific: after observing the offline data, which online actions are worth taking because they cheaply resolve what remains unknown? A diagnostic action can be useful even when its immediate reward is lower than every task-oriented alternative.

Method

The paper measures residual uncertainty through conditional mutual information: how much the online trajectories can reveal about a learning target after conditioning on the offline dataset. Information-directed sampling evaluates an action by its expected regret relative to the information it provides.

The selection rule minimizes Δ² / (g + η), where Δ is instantaneous expected regret, g is expected information gain, and η is a nonnegative regularization parameter. Setting η to zero gives vanilla IDS. A low-reward probe can be selected when its information benefit makes the ratio small.

The analysis first establishes a Bayesian regret bound through a ratio certificate. A reference Thompson-sampling policy supplies a bound that IDS inherits over the same randomized policy class. A known-dynamics Bayesian linear-reward model then makes the remaining information explicit. For deep RL, ROID implements the selection idea using an ensemble and information surrogates. Theory, deep implementation

Key findings

1. An informative probe can separate IDS from probability matching

In the hidden-mode bandit, a probe is informative but is never the optimal action under a sampled model. At offline sample size N = 1000, residual mode probability p_N = 0.0066, and online horizon T = 500, vanilla IDS incurs 0.15 cumulative regret, versus 1.79 for Thompson sampling and 3.31 for greedy or UCB. Results average ten seeds. Lower regret is better. Table 1

Method	Cumulative regret, hidden-mode setting
Greedy / UCB	3.31
Thompson sampling	1.79
Vanilla IDS	0.15

2. The theoretical bound depends on uncertainty left after the warm start

The generic bound uses conditional information after the offline dataset has been observed. In the known-dynamics linear-Gaussian setting, this information has a log-determinant form. The resulting rate depends on a coverage coefficient tied to the visitation distribution induced by vanilla IDS itself. This is an assumption-dependent Bayesian guarantee. Section 4

3. Benefits depend on the remaining bias

In the biased contextual bandit with N = 20, T = 200, and twenty seeds, IDS with η = 0.5 gives 3.57 ± 1.67 cumulative regret, compared with UCB’s 4.80 ± 1.37 and Thompson sampling’s 10.64 ± 1.52. With N = 100, UCB is better: 0.113 ± 0.33, versus 0.641 ± 1.95 for the same IDS variant. More offline data changes which exploration rule helps. Appendix D, Table 4

4. The selection idea also improves a deep RL implementation

ROID has the highest reported online return on four of six D4RL-v2 locomotion tasks. Its summed score is 648.9, compared with 633.2 for BOORL. ROID averages three seeds and uses one million online environment steps. The table below reports normalized online return, not a percentage of successful episodes. Table 2

D4RL-v2 task	BOORL	ROID
Hopper-medium	109.8	113.22
Hopper-medium-replay	111.1	106.33
Walker2d-medium	107.7	110.60
Walker2d-medium-replay	114.4	131.58
HalfCheetah-medium	98.7	95.51
HalfCheetah-medium-replay	91.5	91.65

Relation to prior work

Thompson sampling explores through probability matching: it samples a model and acts optimally for that model. A diagnostic action that is suboptimal in every sampled model can therefore remain unused. IDS explicitly evaluates its information value, allowing that action to be selected when it resolves uncertainty at sufficiently low regret.

BOORL is the closest offline-to-online comparison in this Bayesian perspective. Other approaches use replay balancing, policy constraints, calibrated pessimism, or exploration bonuses to manage the transition. This paper isolates the information that remains after the offline warm start and uses it to guide action selection. Related work

Limitations

The formal guarantees do not extend to ROID’s deep-ensemble posterior or information surrogates. Its results provide empirical evidence for the selection rule on six locomotion tasks, with three seeds, and ROID does not lead on every task.

Regularization is also regime-dependent. In the rare-mode experiment with N = 1000, the tested positive-η variants incur 3.31 regret, while vanilla IDS remains at 0.15. A regularizer that helps one regime can suppress a useful diagnostic probe in another. Section 6, Appendix D, Table 3

Resources

The downloadable citation identifies the available arXiv version; the conference and workshop appearances are recorded separately above.

BibTeX

Download .bib

@misc{chen2026informationdirected,
  title = {Information-Directed Offline-to-Online Reinforcement Learning},
  author = {Chen, Keru},
  year = {2026},
  eprint = {2605.29405},
  archivePrefix = {arXiv},
  primaryClass = {cs.LG},
  doi = {10.48550/arXiv.2605.29405},
  url = {https://arxiv.org/abs/2605.29405}
}