Paper: arXiv 2312.15385

Abstract

This paper studies a discrete-time mean-variance model based on reinforcement learning. Compared with its continuous-time counterpart in \cite{“zhou2020mv”}, the discrete-time model makes more general assumptions about the asset’s return distribution. Using entropy to measure the cost of exploration, we derive the optimal investment strategy, whose density function is also Gaussian type. Additionally, we design the corresponding reinforcement learning algorithm. Both simulation experiments and empirical analysis indicate that our discrete-time model exhibits better applicability when analyzing real-world data than the continuous-time model.

Complexity vs Empirical Score

  • Math Complexity: 8.0/10
  • Empirical Rigor: 6.0/10 (capped from 6.5: transaction costs not modelled; see penalty flags)
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper employs dense stochastic control and reinforcement learning theory, including derivations of Gaussian optimal policies and convergence proofs (high math). It validates the model with both simulation experiments and an empirical backtest on S&P 500 data, demonstrating a focus on real-world applicability (moderate-to-high rigor).

Research Flowchart

  flowchart TD
  A["Research Goal:<br>Discrete-Time Mean-Variance Model<br>Based on Reinforcement Learning"] --> B["Methodology: Derive Optimal Strategy"]
  B --> C["Methodology: Design RL Algorithm<br>with Entropy Regularization"]
  C --> D["Data: Simulation &<br>Empirical Market Data"]
  D --> E["Computation: Apply Model & Algorithm"]
  E --> F["Outcomes: Gaussian Type<br>Optimal Investment Density"]
  E --> G["Outcomes: Better Applicability<br>vs. Continuous-Time Models"]

Cited References (33)

  1. Optimal dynamic portfolio selection: Multiperiod mean-variance formulation (2000)
  2. Continuous-time mean-variance portfolio selection: A stochastic LQ framework (2000)
  3. Continuous-time mean–variance portfolio selection: A reinforcement learning framework (2020)
  4. Portfolio selection (1952)
  5. Reinforcement learning in continuous time and space: A stochastic control approach (2020)
  6. Large scale continuous-time mean-variance portfolio allocation via reinforcement learning (2019)
  7. Financial data and the skewed generalized t distribution (1998)
  8. Mean-variance hedging in continuous time (1991)
and 25 more
  1. Optimal execution of portfolio transactions (2001)
  2. Recent advances in reinforcement learning in finance (2023)
  3. Algorithm trading using Q -learning and recurrent reinforcement learning (2016)
  4. Reinforcement learning in Portfolio Management and its interpretation (2018)
  5. Portfolio Management using Reinforcement Learning (2016)
  6. Algorithmic aspects of mean–variance optimization in Markov decision processes (2013)
  7. Model-free reinforcement learning for financial portfolios: a brief survey (2019) — link
  8. An intelligent financial portfolio trading strategy using deep Q -learning (2020)
  9. G-learner and girl: Goal based wealth management with reinforcement learning (2020) — link
  10. A deep reinforcement learning framework for the financial portfolio management problem (2017) — link
  11. Practical deep reinforcement learning approach for stock trading (2018) — link
  12. Continuous control with deep reinforcement learning (2015) — link
  13. Adversarial deep reinforcement learning in portfolio management (2018) — link
  14. Reinforcement Learning for Continuous-Time Mean-Variance Portfolio Selection in a Regime-Switching Market (2023)
  15. Policy evaluation and temporal-difference learning in continuous time and space: A martingale approach (2022)
  16. Policy gradient and actor-critic learning in continuous time and space: Theory and algorithms (2022)
  17. q-Learning in Continuous Time (2023)
  18. Dynamic mean-variance portfolio selection with no-shorting constraints (2002)
  19. Optimal multi-period mean-variance policy under no-shorting constraint (2014)
  20. Regime-switching recurrent reinforcement learning for investment decision making (2012)
  21. Transition variable selection for regime switching recurrent reinforcement learning (2014)
  22. Adaptive quantitative trading: An imitative deep reinforcement learning approach (2020)
  23. A reinforcement learning approach for solving the mean variance customer portfolio in partially observable models (2018)
  24. Survey on Multi-period Mean–Variance Portfolio Selection Model (2022)
  25. Learning equilibrium mean-variance strategy (2023)

0 of 33 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.