Paper: arXiv 2412.16175

Abstract

We study continuous-time mean–variance portfolio selection in markets where stock prices are diffusion processes driven by observable factors that are also diffusion processes, yet the coefficients of these processes are unknown. Based on the recently developed reinforcement learning (RL) theory for diffusion processes, we present a general data-driven RL algorithm that learns the pre-committed investment strategy directly without attempting to learn or estimate the market coefficients. For multi-stock Black–Scholes markets without factors, we further devise a baseline algorithm and prove its performance guarantee by deriving a sublinear regret bound in terms of the Sharpe ratio. For performance enhancement and practical implementation, we modify the baseline algorithm and carry out an extensive empirical study to compare its performance, in terms of a host of common metrics, with a large number of widely employed portfolio allocation strategies on S&P 500 constituents. The results demonstrate that the proposed continuous-time RL strategy is consistently among the best, especially in a volatile bear market, and decisively outperforms the model-based continuous-time counterparts by significant margins.

Complexity vs Empirical Score

  • Math Complexity: 8.5/10
  • Empirical Rigor: 7.5/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper employs advanced continuous-time stochastic analysis and proves sublinear regret bounds, indicating high mathematical density. It also features an extensive empirical study with real S&P 500 data, comparing multiple strategies across various metrics, demonstrating substantial implementation and backtesting rigor.

Research Flowchart

  flowchart TD
  A["Research Goal:<br>Mean-Variance Portfolio<br>Selection in Unknown Markets"]
  
  B["Methodology:<br>Continuous-Time<br>Reinforcement Learning RL"]
  
  C["Inputs:<br>S&P 500 Constituents<br>Real Market Data"]
  
  D["Computational Process:<br>Direct Strategy Learning<br>No Model Estimation"]
  
  E["Key Findings:<br>Sublinear Regret Bound<br>& Superior Sharpe Ratio"]
  
  F["Outcomes:<br>Consistent Top Performance<br>Especially in Bear Markets"]
  
  A --> B
  B --> C
  C --> D
  D --> E
  E --> F

Cited References (166)

  1. Who are I : Time inconsistency and intrapersonal conflict and reconciliation (2022)
  2. Data-Driven M erton’s Strategies via Policy Randomization — → 2312.11797
  3. Improved estimation of the covariance matrix of stock returns with an application to portfolio selection (2003)
  4. Optimal versus naive diversification: How inefficient is the 1/N portfolio strategy? (2009)
  5. Portfolio selection (1952)
  6. Bayes- S tein estimation for portfolio analysis (1986)
  7. Common risk factors in the returns on stocks and bonds (1993)
  8. Asset allocation: Combining investor views with market equilibrium (1990)
and 158 more
  1. Distributionally robust mean-variance portfolio selection with Wasserstein distances (2022)
  2. Capital asset prices: A theory of market equilibrium under conditions of risk (1964)
  3. Risk reduction in large portfolios: Why imposing the wrong constraints helps (2003)
  4. Estimation with quadratic loss (1992)
  5. Minimum-variance portfolios in the US equity market (2006)
  6. A simplified model for portfolio analysis (1963)
  7. Individual comparisons by ranking methods (1992)
  8. Policy evaluation and temporal-difference learning in continuous time and space: A martingale approach (2022)
  9. Policy gradient and actor-critic learning in continuous time and space: Theory and algorithms (2022)
  10. Reinforcement learning in continuous time and space (2000)
  11. Reinforcement learning in continuous time and space: A stochastic control approach (2020)
  12. Continuous-time mean–variance portfolio selection: A reinforcement learning framework (2020)
  13. Stochastic Controls: Hamiltonian Systems and HJB Equations (1999)
  14. Controlled Markov Processes and Viscosity Solutions (2006)
  15. Investment Science (1998)
  16. A closed-form solution for options with stochastic volatility with applications to bond and currency options (1993)
  17. Smart “predict, then optimize” (2022)
  18. Optimal dynamic portfolio selection: Multiperiod mean-variance formulation (2000)
  19. Continuous-time mean-variance portfolio selection: A stochastic LQ framework (2000)
  20. Portfolio selection in stochastic environments (2007)
  21. Markowitz’s mean-variance portfolio selection with regime switching: A continuous-time model (2003)
  22. Numerical solution of the Hamilton–Jacobi–Bellman formulation for continuous-time mean–variance asset allocation under stochastic volatility (2016)
  23. Mean–variance portfolio selection of cointegrated assets (2011)
  24. Mean-variance portfolio selection with random parameters in a complete market (2002)
  25. A dynamic mean-variance analysis for log returns (2021)
  26. Dynamic mean-variance asset allocation (2010)
  27. Portfolio and consumption decisions under mean-reverting returns: An exact solution for complete markets (2002)
  28. On estimating the expected return on the market: An exploratory investigation (1980)
  29. On the sensitivity of mean-variance-efficient portfolios to changes in asset means: Some analytical and computational results (1991)
  30. The sampling error in estimates of mean-variance efficient portfolio weights (1999)
  31. On portfolio optimization: Forecasting covariances and choosing the risk model (1999)
  32. A generalized approach to portfolio optimization: Improving performance by constraining portfolio norms (2009)
  33. The effect of errors in means, variances, and covariances on optimal portfolio choice (2013)
  34. Curse of optimality, and how to break it? (2023)
  35. Mean–Variance Optimization for Asset Allocation (2021)
  36. On the existence of optimal relaxed controls of stochastic partial differential equations (1992)
  37. On stochastic relaxed control for partially observed diffusions (1984)
  38. Learning equilibrium mean-variance strategy (2023)
  39. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor (2018)
  40. Maximum entropy inverse reinforcement learning. (2008)
  41. Sensitivity analysis for mean-variance portfolio problems (1991)
  42. Dynamic Nonmyopic Portfolio Behavior (1996)
  43. Consumption and portfolio decisions when expected returns are time varying (1999)
  44. Mastering the game of Go without human knowledge (2017)
  45. Human-level control through deep reinforcement learning (2015)
  46. End-to-end training of deep visuomotor policies (2016)
  47. A survey on policy search for robotics (2013)
  48. Estimating Portfolio and Consumption Choice: A Conditional E uler Equations Approach (1999)
  49. Vast portfolio selection with gross-exposure constraints (2012)
  50. Variable Selection for Portfolio Choice (2001)
  51. Strategic Asset Allocation: Portfolio Choice for Long-Term Investors (2002)
  52. Continuous-time Markowitz’s model with transaction costs (2010)
  53. Continuous-time mean-variance efficiency: the 80% rule (2006)
  54. A theory of Markovian time-inconsistent stochastic control in discrete time (2014)
  55. Mean–variance portfolio optimization with state-dependent risk aversion (2014)
  56. Myopia and inconsistency in dynamic utility maximization (1955)
  57. Being serious about non-commitment: Subgame perfect equilibrium in continuous time (2006)
  58. Time-consistent portfolio management (2012)
  59. Consumption and portfolio rules for time-inconsistent investors (2010)
  60. Investment and consumption without commitment (2008)
  61. The golden rule when preferences are time inconsistent (2010)
  62. Time-inconsistent stochastic linear-quadratic control: characterization and uniqueness of equilibrium (2017)
  63. Time-inconsistent stochastic linear–quadratic control (2012)
  64. Time-inconsistent recursive stochastic optimal control problems (2017)
  65. Time-inconsistent optimal control problems and the equilibrium HJB equation (2012) — link
  66. Mean-variance portfolio selection with dynamic targets for expected terminal wealth (2022)
  67. Portfolio analysis under uncertain means, variances, and covariances (1974)
  68. Inadmissibility of the usual estimator for the mean of a multivariate normal distribution (1956)
  69. Portfolio selection with parameter and model uncertainty: A multi-prior approach (2007)
  70. Robust portfolio selection problems (2003)
  71. Maxmin expected utility with a non-unique prior (1989)
  72. Portfolio diversification and model uncertainty: A robust dynamic mean-variance approach (2022)
  73. Robust Markowitz mean-variance portfolio selection under ambiguous covariance matrix (2019)
  74. Machine learning and portfolio optimization (2018)
  75. Robust satisficing (2023)
  76. Nonlinear shrinkage of the covariance matrix for portfolio selection: Markowitz meets Goldilocks (2017)
  77. High dimensional covariance matrix estimation using a factor model (2008)
  78. FinBrain: when finance meets AI 2.0 (2019)
  79. Portfolio management using reinforcement learning (2016)
  80. An algorithm for trading and portfolio management using Q-learning and Sharpe ratio maximization (2000)
  81. Optimal asset allocation using adaptive dynamic programming (1996)
  82. Machine learning for trading (2017)
  83. Algorithmic aspects of mean–variance optimization in Markov decision processes (2013)
  84. The variance of discounted Markov decision processes (1982)
  85. TD algorithm for the variance of return and mean-variance reinforcement learning (2001)
  86. Variance-penalized reinforcement learning for risk-averse asset allocation (2000)
  87. Variance adjusted actor critic algorithms (2013) — link
  88. Variance-constrained actor-critic algorithms for discounted and average reward MDPs (2016)
  89. Actor-critic algorithms for risk-sensitive MDPs (2013)
  90. Reinforcement Learning: An Introduction (2018)
  91. A stochastic approximation algorithm with varying bounds (1995)
  92. General bounds and finite-time improvement for the Kiefer-Wolfowitz stochastic approximation algorithm (2011)
  93. A stochastic approximation method (1951)
  94. Stochastic Approximation (2003)
  95. Stochastic Approximation: A Dynamical Systems Viewpoint (2008) — link
  96. An overview of stochastic approximation (2014)
  97. Generalized instrumental variables estimation of nonlinear rational expectations models (1982)
  98. A practical end-to-end inventory management model with deep learning (2023)
  99. Risk parity and diversification (2011)
  100. Achieving mean–variance efficiency by continuous-time reinforcement learning (2022)
  101. On the relation between several discrete choice models (2017)
  102. Empirical asset pricing via machine learning (2020)
  103. Autoencoder asset pricing models (2021)
  104. Machine learning in the Chinese stock market (2022)
  105. Deep learning in asset pricing (2024)
  106. Bond risk premiums with machine learning (2021)
  107. (Re-) Imag (in) ing price trends (2020)
  108. Deep learning statistical arbitrage (2021) — link
  109. A convergence theorem for non negative almost supermartingales and some applications (1971)
  110. Numerical Solution of Stochastic Differential Equations (1992)
  111. Continuous control with deep reinforcement learning (2016) — link
  112. Proximal Policy Optimization Algorithms (2017) — link
  113. High-Dimensional Continuous Control Using Generalized Advantage Estimation (2016) — link
  114. Factors that fit the time series and cross-section of stock returns (2020)
  115. Conditional choice probabilities and the estimation of dynamic models (1993)
  116. Stochastic choice and revealed perturbed utility (2015)
  117. Machine learning in asset management—part 2: Portfolio construction—weight optimization (2020)
  118. Machine Learning for Active Portfolio Management (2021)
  119. Dynamic portfolio choice with parameter uncertainty and the economic value of analysts’ recommendations (2006)
  120. Optimal portfolio choice under incomplete information (1986)
  121. Empirical asset pricing: The cross section of stock returns (2016)
  122. The cross-section of expected stock returns (2015)
  123. Presidential address: Discount rates (2011)
  124. Evidence of predictable behavior of security returns (1990)
  125. Fads, martingales, and market efficiency (1990)
  126. Returns to buying winners and selling losers: Implications for stock market efficiency (1993)
  127. Predicting excess stock returns out of sample: Can anything beat the historical average? (2008)
  128. Policy gradient in continuous time (2006)
  129. Making deep Q -learning methods robust to time discretization (2019)
  130. Time discretization-invariant safe action repetition for policy gradient methods (2021)
  131. The properties of equally weighted risk contribution portfolios (2010)
  132. Reinforcement learning for jump-diffusions, with financial applications — → 2405.16449
  133. Estimating diffusions with discretely and possibly randomly spaced data: A general theory (2004)
  134. Likelihood inference for diffusions: A survey (2006)
  135. Maximum likelihood estimation of stochastic volatility models (2007)
  136. Maximum likelihood estimation of discretely sampled diffusions: A closed-form approximation approach (2002)
  137. Back to the Future: Generating Moment Implications for Continuous-Time Markov Processes (1995)
  138. Maximum likelihood estimation of generalized It ^o processes with discretely sampled data (1988)
  139. MCMC analysis of diffusion models with application to finance (2001)
  140. Estimating equations based on eigenfunctions for a discretely observed diffusion process (1999)
  141. Closed-Form Likelihood Expansions for Multivariate Diffusions (2008)
  142. A nonparametric model of term structure dynamics and the market price of interest rate risk (1997)
  143. The limits to learning a diffusion model (2021)
  144. Accuracy of discretely sampled stochastic policies in continuous-time reinforcement learning (2025) — link
  145. Regret of exploratory policy improvement and $q$-learning (2024) — link
  146. Square-root regret bounds for continuous-time episodic M arkov decision processes (2025)
  147. Unifying PAC and regret: Uniform PAC bounds for episodic reinforcement learning (2017)
  148. Optimal scheduling of entropy regularizer for continuous-time linear-quadratic reinforcement learning (2024)
  149. Optimistic Q -learning for average reward and episodic reinforcement learning (2024) — link
  150. Is Q -learning provably efficient? (2018)
  151. Breaking the sample complexity barrier to regret-optimal model-free reinforcement learning (2021)
  152. Provably efficient reinforcement learning with linear function approximation (2023)
  153. Risk-sensitive reinforcement learning: Near-optimal risk-sample tradeoff in regret (2020)
  154. Exponential B ellman equation and improved regret bounds for risk-sensitive reinforcement learning (2021)
  155. Distributionally Robust $ Q $-Learning (2022)
  156. A finite sample complexity bound for distributionally robust Q -learning (2023)
  157. Regret Bounds for Episodic Risk-Sensitive Linear Quadratic Regulator (2025)
  158. Regret bounds for M arkov decision processes with recursive optimized certainty equivalents (2023)

2 of 166 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.