Paper: arXiv 2501.16659

Abstract

Considering the continuous-time Mean-Variance (MV) portfolio optimization problem, we study a regime-switching market setting and apply reinforcement learning (RL) techniques to assist informed exploration within the control space. We introduce and solve the Exploratory Mean Variance with Regime Switching (EMVRS) problem. We also present a Policy Improvement Theorem. Further, we recognize that the widely applied Temporal Difference (TD) learning is not adequate for the EMVRS context, hence we consider Orthogonality Condition (OC) learning, leveraging the martingale property of the induced optimal value function from the analytical solution to EMVRS. We design a RL algorithm that has more meaningful parameterization using the market parameters and propose an updating scheme for each parameter. Our empirical results demonstrate the superiority of OC learning over TD learning with a clear convergence of the market parameters towards their corresponding ``grounding true" values in a simulated market scenario. In a real market data study, EMVRS with OC learning outperforms its counterparts with the highest mean and reasonably low volatility of the annualized portfolio returns.

Complexity vs Empirical Score

  • Math Complexity: 9.5/10
  • Empirical Rigor: 6.0/10 (capped from 7.0: transaction costs not modelled; see penalty flags)
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper uses advanced continuous-time stochastic control, Lagrangian duality, and RL theory, indicating very high math complexity. It includes both simulated convergence tests and real-market backtests with performance metrics, showing strong empirical rigor.

Research Flowchart

  flowchart TD
  A["Research Goal: How to solve MV Portfolio Optimization<br>with Regime Switching (EMVRS) using RL?"] --> B["Methodology: Exploratory MV with Regime Switching<br>Policy Improvement Theorem"]
  B --> C["Key Innovation:<br>Orthogonality Condition (OC) Learning<br>vs Standard TD Learning"]
  C --> D["Data & Simulation:<br>Simulated Markets with 'True' Parameters"]
  D --> E["Computational Process:<br>RL Algorithm with Market Parameter Updates"]
  E --> F["Key Findings:<br>1. OC Learning converges to true parameters<br>2. OC outperforms TD in accuracy<br>3. Real market: Highest returns & low volatility"]

Cited References (88)

  1. Continuous-time mean–variance portfolio selection: A reinforcement learning framework (2020)
  2. Continuous-time mean-variance portfolio selection: A stochastic LQ framework (2000)
  3. Reinforcement learning in continuous time and space: A stochastic control approach (2020)
  4. The reinforcement learning kelly strategy (2022)
  5. Policy evaluation and temporal-difference learning in continuous time and space: A martingale approach (2022) — link
  6. Asset and liability management under a continuous-time mean–variance optimization framework (2006)
  7. Continuous-time portfolio selection with liability: Mean–variance model and stochastic LQ approach (2008)
  8. Optimal dynamic portfolio selection: Multiperiod mean-variance formulation (2000)
and 80 more
  1. q-Learning in continuous time (2023)
  2. Control randomisation approach for policy gradient and application to reinforcement learning in optimal switching (2024) — link
  3. Learning equilibrium mean-variance strategy (2023)
  4. Policy gradient and actor-critic learning in continuous time and space: Theory and algorithms (2022)
  5. Reinforcement learning for continuous-time mean-variance portfolio selection in a regime-switching market (2024)
  6. Portfolio selection (1952)
  7. Adaptive Control Processes-A Guided Tour (1961)
  8. Dynamic programming (1966)
  9. A stochastic reinforcement learning algorithm for learning real-valued functions (1990)
  10. Pattern-recognizing stochastic learning automata (1985)
  11. I. On governors (1868)
  12. Q-learning (1992)
  13. Policy gradient methods for reinforcement learning with function approximation (1999)
  14. PORTFOLIO SELECTION* (1952) — link
  15. Stochastic adaptive control methods: a survey (1975)
  16. A survey of some results in stochastic adaptive control (1985)
  17. A survey of numerical methods for optimal control (2009)
  18. Recent developments in machine learning methods for stochastic control and games (2023) — link
  19. Portfolio selection problems with markowitz’s mean–variance framework: a review of literature (2018)
  20. A comprehensive review of deterministic models and applications for mean-variance portfolio optimization (2019)
  21. Reinforcement learning: A survey (1996)
  22. Learning to predict by the methods of temporal differences (1988)
  23. Deep reinforcement learning: A brief survey (2017)
  24. Model-free reinforcement learning for financial portfolios: a brief survey (2019) — link
  25. Risk-Averse Bayes-Adaptive Reinforcement Learning (2021)
  26. Efficient Risk-Averse Reinforcement Learning (2022)
  27. Risk-Averse Reinforcement Learning for Portfolio Optimization (2021)
  28. Risk-Averse Distributional Reinforcement Learning: A CVaR Optimization Approach (2019) — link
  29. Mean-variance policy iteration for risk-averse reinforcement learning (2021)
  30. Reinforcement Learning with Non-Exponential Discounting (2022)
  31. Delayed Geometric Discounts: An Alternative Criterion for Reinforcement Learning (2022) — link
  32. UGAE : A Novel Approach to Non-exponential Discounting (2023) — link
  33. Average reward adjusted discounted reinforcement learning: Near-blackwell-optimal policies for real-world applications (2020) — link
  34. A Subgame Perfect Equilibrium Reinforcement Learning Approach to Time-inconsistent Problems (2021) — link
  35. Time-inconsistent objectives in reinforcement learning (2021)
  36. Reinventing Policy Iteration under Time Inconsistency (2022) — link
  37. Markov Decision Processes with Time-Varying Geometric Discounting (2023)
  38. Markowitz’s mean-variance portfolio selection with regime switching: A continuous-time model (2003)
  39. Markowitz’s mean-variance portfolio selection with regime switching: From discrete-time models to their continuous-time limits (2004)
  40. Markowitz’s mean-variance asset–liability management with regime switching: A multi-period model (2011)
  41. Markowitz’s mean-variance asset-liability management with regime switching: A continuous-time model (2008)
  42. Multi-period mean-variance portfolio selection with markov regime switching and uncertain time-horizon (2011)
  43. Multi-period markowitz’s mean–variance portfolio selection with state-dependent exit probability (2014)
  44. Explicit solutions of consumption-investment problems in financial markets with regime switching (2009)
  45. A stochastic control problem with regime switching (2021)
  46. Asset allocation under stochastic interest rate with regime switching (2012)
  47. Classical and singular stochastic control for the optimal dividend policy when there is regime switching (2011)
  48. Stochastic optimal control for nonlinear Markov jump diffusion processes (2012)
  49. Identifying bull and bear markets in stock returns (2000)
  50. International asset allocation with regime shifts (2002)
  51. How regimes affect asset allocation (2004)
  52. Dynamic asset allocation for varied financial markets under regime switching framework (2014)
  53. Neural Ordinary Differential Equations (2018)
  54. Attention Is All You Need (2017) — link
  55. A hidden markov model variant for sequence classification (2011)
  56. Similarity-based classification of sequences using hidden markov models (2004)
  57. Improving time series classification using hidden markov models (2012)
  58. The viterbi algorithm (1973)
  59. Error bounds for convolutional codes and an asymptotically optimum decoding algorithm (1967) — link
  60. Partially observable markov decision processes (2012)
  61. Continuous-time stochastic control and optimization with financial applications (2009)
  62. Stochastic control and application to finance (2002)
  63. Machine learning in Finance (2020)
  64. Dynamic programming and optimal control: Volume I (2012)
  65. Dynamic programming and optimal control 4th edition, volume ii (2015)
  66. Partial differential equations with numerical methods (2003)
  67. The behavior of organisms: An experimental analysis (1938)
  68. Investment Science (2013)
  69. The Econometrics of Financial Markets (1997)
  70. A general theory of Markovian time inconsistent stochastic control problems (2010)
  71. A theory of Markovian time-inconsistent stochastic control in discrete time (2014)
  72. A theory of markovian time inconsistent stochastic control in continuous time (2016)
  73. Time-Inconsistent Control Theory with Finance Applications (2021)
  74. PyTorch: An Imperative Style, High-Performance Deep Learning Library (2019) — link
  75. Utility Theory for Decision Making (1979) — link
  76. Anomalies: Utility maximization and experienced utility (2006)
  77. Anomalies: preference reversals (1990)
  78. Playing Atari with Deep Reinforcement Learning (2013) — link
  79. A deep reinforcement learning framework for the financial portfolio management problem (2017) — link
  80. Reinforcement learning for options trading (2021)

0 of 88 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.