Paper: arXiv 2512.04697

Authors: Yijie Huang, Mengge Li, Xiang Yu, Zhou Zhou

Abstract

This paper studies the continuous-time reinforcement learning (RL) for optimal switching problems across multiple regimes. We consider a type of exploratory formulation under entropy regularization where the agent randomizes both the timing of switches and the selection of regimes through the generator matrix of an associated continuous-time finite-state Markov chain. We establish the well-posedness of the associated system of Hamilton-Jacobi-Bellman (HJB) equations and provide a characterization of the optimal policy. The policy improvement and the convergence of the policy iterations are rigorously established by analyzing the system of equations. We also show the convergence of the value function in the exploratory formulation towards the value function in the classical formulation as the temperature parameter vanishes. Finally, a reinforcement learning algorithm is devised and implemented by invoking the policy evaluation based on the martingale characterization. Our numerical examples with the aid of neural networks illustrate the effectiveness of the proposed RL algorithm.

Complexity vs Empirical Score

  • Math Complexity: 9.5/10
  • Empirical Rigor: 4.0/10
  • Quadrant: Lab Rats — theoretically deep, empirically untested

Why this score: The paper is mathematically dense, featuring advanced stochastic analysis, systems of Hamilton-Jacobi-Bellman (HJB) equations, and rigorous convergence proofs. While it includes a numerical implementation with neural networks, the core contribution is theoretical with analytical proofs dominating the rigor, focusing on PDE existence and convergence rather than extensive backtesting on real financial datasets.

Research Flowchart

  flowchart TD
  A["Research Goal<br>Continuous-time RL for Optimal Switching"] --> B["Methodology: Entropy-Regularized Formulation"]
  B --> C{"Analytical Derivation"}
  C --> D["HJB Equations &<br>Optimal Policy Characterization"]
  C --> E["Convergence Analysis<br>Exploratory → Classical Value"]
  B --> F["Algorithm Design<br>RL with Policy Evaluation"]
  F --> G["Computational Process<br>Neural Network Implementation"]
  D & E & G --> H["Outcomes<br>Validated RL Algorithm &<br>Theoretical Guarantees"]

Cited References (39)

  1. A stochastic approximation method (1951)
  2. Hybrid Switching Diffusions: Properties and Applications (2025)
  3. Continuous time q-learning for mean-field control problems — → 2306.16208
  4. On optimal tracking portfolio in incomplete markets: T he reinforcement learning approach (2025)
  5. Reinforcement learning for continuous-time mean-variance portfolio selection in a regime-switching market (2024)
  6. A reinforcement learning framework for some singular stochastic control problems (2025) — link
  7. Reinforcement learning for irreversible reinsurance problems: the randomized singular control approach (2025) — link
  8. Continuous-time q-learning for jump-diffusion models under T sallis entropy (2026)
and 31 more
  1. Reinforcement learning for jump-diffusions, with financial applications — → 2405.16449
  2. Exploratory optimal stopping: A singular control formulation — → 2408.09335
  3. Learning to optimally stop diffusion processes, with financial applications — → 2408.09242
  4. Exploratory HJB equations and their convergence (2022)
  5. Policy iteration for exploratory H amilton– J acobi– B ellman equations (2025)
  6. Convergence of policy iteration for entropy-regularized stochastic control problems (2025)
  7. q- L earning in continuous time (2023)
  8. Policy gradient and actor-critic learning in continuous time and space: T heory and algorithms (2022)
  9. Reinforcement learning for arbitrage strategies in stock index futures (2025)
  10. A two-fold randomization framework for impulse control problems (2025) — link
  11. Control randomisation approach for policy gradient and application to reinforcement learning in optimal switching (2025)
  12. Reinforcement learning in continuous time and space: A stochastic control approach (2020)
  13. Valuation of energy storage: A n optimal switching approach (2010)
  14. Valuation of a power plant under production constraints and market incompleteness (2009)
  15. Management strategies for run-of-river hydropower plants: an optimal switching approach (2022)
  16. Pricing asset scheduling flexibility using optimal switching (2008)
  17. Adaptive algorithms and stochastic approximations (2012)
  18. A stochastic target formulation for optimal switching problems in finite horizon (2009)
  19. Policy evaluation and temporal-difference learning in continuous time and space: A martingale approach (2022)
  20. Optimal control and viscosity solutions of H amilton- J acobi- B ellman equations (1997)
  21. Stochastic optimal multi-modes switching with a viscosity solution approach (2013)
  22. On the first boundary problem for quasilinear systems of parabolic differential equations in non-cylindrical domains (1965)
  23. Continuous-time stochastic control and optimization with financial applications (2009)
  24. Explicit solution to an optimal switching problem in the two‐regime case (2007) — link
  25. Exploratory optimal stopping: A singular control formulation — → 2408.09335
  26. Randomized optimal stopping problem in continuous time and reinforcement learning algorithm (2024)
  27. Policy gradient and actor-critic learning in continuous time and space: T heory and algorithms (2022)
  28. Unified continuous-time q-learning for mean-field game and mean-field control problems — → 2407.04521
  29. Regret of exploratory policy improvement and q-learning (2024) — link
  30. Continuous-time q-learning for mean-field control with common noise, part-I: theoretical foundations (2026) — link
  31. Continuous-time q-learning for mean-field control with common noise, part-II: q-learning algorithms (2026) — link

6 of 39 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.