Paper: arXiv 2411.12746

Abstract

In recent years, there has been a growing trend of applying Reinforcement Learning (RL) in financial applications. This approach has shown great potential to solve decision-making tasks in finance. In this survey, we present a comprehensive study of the applications of RL in finance and conduct a series of meta-analyses to investigate the common themes in the literature, such as the factors that most significantly affect RL’s performance compared to traditional methods. Moreover, we identify challenges including explainability, Markov Decision Process (MDP) modeling, and robustness that hinder the broader utilization of RL in the financial industry and discuss recent advancements in overcoming these challenges. Finally, we propose future research directions, such as benchmarking, contextual RL, multi-agent RL, and model-based RL to address these challenges and to further enhance the implementation of RL in finance.

Complexity vs Empirical Score

  • Math Complexity: 7.0/10
  • Empirical Rigor: 3.0/10
  • Quadrant: Lab Rats — theoretically deep, empirically untested

Why this score: The paper reviews mathematical concepts like Markov Decision Processes and RL algorithms (Policy Gradient, PPO) with theoretical categorizations, but it is a survey paper with meta-analysis of existing studies rather than presenting new empirical backtests or implementation-heavy data, resulting in lower empirical rigor.

Research Flowchart

  flowchart TD
  A["Research Goal: Review RL in Finance"] --> B["Methodology: Meta-Analysis of Literature<br>(Factors affecting RL performance)"]
  B --> C["Data/Inputs: Financial Applications<br>(Trading, Portfolio, Risk Mgmt)"]
  C --> D["Computational Process: MDP Modeling &<br>RL Algorithm Evaluation"]
  D --> E["Key Findings: Challenges<br>(Explainability, Robustness)"]
  E --> F["Outcomes: Future Directions<br>(Benchmarking, Multi-Agent, Contextual RL)"]

Cited References (155)

  1. Modelling under-reporting in epidemics (2014)
  2. Estimation of under-reporting in epidemics using approximations (2017)
  3. A Monte Carlo approach to nonnormal and nonlinear state-space modeling (1992)
  4. Discretely Observed B rownian Motion Governed by Telegraph Process: E stimation (2019) — link
  5. The L aplace Distribution and Generalizations: A Revisit with Applications to Communications, Economics, Engineering, and Finance (2001)
  6. tensortrade package (2024)
  7. RLlib: Abstractions for distributed reinforcement learning (2018)
  8. Stable-baselines3: Reliable reinforcement learning implementations (2021)
and 147 more
  1. Tensorflow agents: Efficient batched reinforcement learning in tensorflow (2017) — link
  2. Reinforcement learning for market making in a multi-agent dealer market (2019) — link
  3. Multi-agent reinforcement learning in a realistic limit order book market simulation (2020)
  4. Finite-time analysis of the multiarmed bandit problem (2002)
  5. Portfolio Selection (1952)
  6. Risk-sensitive Markov control processes (2013)
  7. Universal trading for order execution with oracle policy distillation (2021)
  8. Risk-averse reinforcement learning for algorithmic trading (2014)
  9. Optimal control of execution costs (1998)
  10. Reinforcement learning for optimized trade execution (2006)
  11. Risk-aware multi-armed bandit problem with application to portfolio selection (2017)
  12. Adaptive portfolio by solving multi-armed bandit via thompson sampling (2019) — link
  13. A tutorial on thompson sampling (2018)
  14. Online portfolio selection: A survey (2014)
  15. Optimal execution of portfolio transactions (2001)
  16. A reinforcement learning extension to the Almgren-Chriss framework for optimal trade execution (2014)
  17. Behavioural and neural characterization of optimistic reinforcement learning (2017)
  18. Portfolio Blending via Thompson Sampling. (2016)
  19. Portfolio choices with orthogonal bandit learning (2015)
  20. An overview of online learning (2004)
  21. Bandit algorithms (2020)
  22. Towards Inverse Reinforcement Learning for Limit Order Book Dynamics (2019) — link
  23. Model-based reinforcement learning for predictions and control for limit order books (2019) — link
  24. On the theory of risk aversion (1970)
  25. Risk aversion in the small and in the large (1978)
  26. Optimizing market making using multi-agent reinforcement learning (2018) — link
  27. Market making via reinforcement learning (2018)
  28. Robust market making via adversarial reinforcement learning (2020) — link
  29. Recent advances in reinforcement learning in finance (2023)
  30. Offline reinforcement learning: Tutorial, review, and perspectives on open problems (2020) — link
  31. Time-driven feature-aware jointly deep reinforcement learning for financial signal representation and algorithmic trading (2020)
  32. A parallel multi-module deep reinforcement learning algorithm for stock trading (2021)
  33. An application of deep reinforcement learning to algorithmic trading (2021)
  34. Adaptive quantitative trading: An imitative deep reinforcement learning approach (2020)
  35. Using reinforcement learning in the algorithmic trading problem (2019)
  36. Returns to Buying Winners and Selling Losers: Implications for Stock Market Efficiency (1993) — link
  37. Deep reinforcement learning for trading (2020)
  38. A deep reinforcement learning framework for the financial portfolio management problem (2017) — link
  39. Reinforcement-learning based portfolio management with augmented asset movement prediction states (2020)
  40. Adversarial deep reinforcement learning in portfolio management (2018) — link
  41. Model-based deep reinforcement learning for dynamic portfolio optimization (2019) — link
  42. Towards principled methods for training generative adversarial networks (2017) — link
  43. Deep deterministic portfolio optimization (2020)
  44. Market making with signals through deep reinforcement learning (2021)
  45. Policy gradient methods for reinforcement learning with function approximation (1999)
  46. Reinforcement learning: An introduction (2018)
  47. Asynchronous methods for deep reinforcement learning (2016)
  48. Playing atari with deep reinforcement learning (2013) — link
  49. Proximal policy optimization algorithms (2017) — link
  50. Reinforcement Learning in Possibly Nonstationary Environments (2022) — link
  51. DeepTrader: A Deep Reinforcement Learning Approach for Risk-Return Balanced Portfolio Management with Market Conditions Embedding (2021)
  52. Applications of Reinforcement Learning in Automated Market-Making (2019)
  53. Commission fee is not enough: A hierarchical reinforced framework for portfolio management (2021)
  54. An electronic market-maker (2001)
  55. Action branching architectures for deep reinforcement learning (2018)
  56. Optimal Market Making by Reinforcement Learning (2021) — link
  57. High-frequency trading in a limit order book (2008)
  58. Deep Reinforcement Learning and Convex Mean-Variance Optimisation for Portfolio Management (2022) — link
  59. Reinforcement Learning Approaches to Optimal Market Making (2021)
  60. ABIDES: Towards high-fidelity multi-agent market simulation (2020)
  61. Offline reinforcement learning: Tutorial, review, and perspectives on open problems (2020) — link
  62. Deep direct reinforcement learning for financial signal representation and trading (2016)
  63. Double deep q-learning for optimal execution (2021)
  64. Improving financial trading decisions using deep Q-learning: Predicting the number of shares, action strategies, and transfer learning (2019)
  65. Multi-agent deep reinforcement learning for liquidation strategy analysis (2019) — link
  66. Deep execution-value and policy based reinforcement learning for trading and beating market benchmarks (2019)
  67. Pattern transfer learning for reinforcement learning in order dispatching (2021) — link
  68. Optimal trade execution based on deep deterministic policy gradient (2020)
  69. Distributional Reinforcement Learning for Optimal Execution
  70. A deep reinforcement learning framework for optimal trade execution (2020)
  71. Learning a functional control for high-frequency finance (2020) — link
  72. An End-to-End Optimal Trade Execution Framework based on Proximal Policy Optimization. (2020)
  73. Policy gradient methods for the noisy linear quadratic regulator over a finite horizon (2021)
  74. FinRL: A deep reinforcement learning library for automated stock trading in quantitative finance (2020) — link
  75. FinRL: Deep reinforcement learning framework to automate trading in quantitative finance (2021)
  76. FinRL-Meta: A Universe of Near-Real Market Environments for Data-Driven Deep Reinforcement Learning in Quantitative Finance (2021) — link
  77. Towards a fully RL-based Market Simulator (2021)
  78. Introduction to Mathematical Finance (2018)
  79. Sim-to-real transfer in deep reinforcement learning for robotics: a survey (2020)
  80. Does the markov decision process fit the data: Testing for the markov property in sequential decision making (2020)
  81. Non-stationarity in financial time series: Generic features and tail behavior (2013)
  82. Analysis of non-stationary dynamics in the financial system (2013)
  83. The elements of financial econometrics (2017)
  84. Learning to trade in financial time series using high-frequency through wavelet transformation and deep reinforcement learning (2021)
  85. Portfolio management using reinforcement learning (2016)
  86. Reinforcement learning algorithm for non-stationary environments (2020)
  87. No-regret reinforcement learning with heavy-tailed rewards (2021)
  88. Detecting and adapting to crisis pattern with context based Deep Reinforcement Learning (2021)
  89. Learning compact models for planning with exogenous processes (2020)
  90. Comprehensive review of deep reinforcement learning methods and applications in economics (2020)
  91. Reinforcement learning in economics and finance (2021)
  92. Deep reinforcement learning in quantitative algorithmic trading: A review (2021) — link
  93. Reinforcement learning in financial markets-a survey (2018)
  94. Reinforcement learning in financial markets (2019)
  95. Deep reinforcement learning for trading—A critical survey (2021)
  96. Deep learning in finance and banking: A literature review and classification (2020)
  97. Financial risk and heavy tails (2003)
  98. Reinforcement learning for high-frequency market making (2018)
  99. Deep reinforcement learning for market making in corporate bonds: beating the curse of dimensionality (2019)
  100. Gaussian based non-linear function approximation for reinforcement learning (2021)
  101. Can Machines’ Learn’Finance? (2020)
  102. A Survey on Interpretable Reinforcement Learning (2021) — link
  103. Explainable Reinforcement Learning: A Survey and Comparative Review (2023) — link
  104. Dynamic programming (1966)
  105. Testing for cointegration with temporally aggregated and mixed‐frequency time series (2015)
  106. Universal value function approximators (2015)
  107. Self-paced deep reinforcement learning (2020)
  108. Self-paced context evaluation for contextual reinforcement learning (2021)
  109. Contextual markov decision processes (2015) — link
  110. Carl: A benchmark for contextual and adaptive reinforcement learning (2021) — link
  111. Robust adversarial reinforcement learning (2017)
  112. Deep reinforcement learning from self-play in imperfect-information games (2016) — link
  113. Data-efficient hierarchical reinforcement learning (2018)
  114. Feudal networks for hierarchical reinforcement learning (2017)
  115. Hierarchical reinforcement learning for multi-agent moba game (2019) — link
  116. Portfolio Selection (1952) — link
  117. A new approach to linear filtering and prediction problems (1960)
  118. Equilibrium points in n-person games (1950)
  119. Efficient capital markets: A review of theory and empirical work (1970)
  120. Risk-sensitive reinforcement learning (2002)
  121. The microeconomics of market making (1986)
  122. Bridging the gap between Markowitz planning and deep reinforcement learning (2020) — link
  123. Model-based value estimation for efficient model-free reinforcement learning (2018) — link
  124. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning (2018)
  125. Practical deep reinforcement learning approach for stock trading (2018) — link
  126. Alphastock: A buying-winners-and-selling-losers investment strategy using interpretable deep reinforcement attention networks (2019)
  127. Optimistic bull or pessimistic bear: Adaptive deep reinforcement learning for stock portfolio allocation (2019) — link
  128. An intelligent financial portfolio trading strategy using deep Q-learning (2020)
  129. Dueling Network Architectures for Deep Reinforcement Learning (2015) — link
  130. Prioritized Experience Replay (2015) — link
  131. Application of deep q-network in portfolio management (2020)
  132. Market sentiment-aware deep reinforcement learning approach for stock portfolio allocation (2021)
  133. Deep reinforcement learning for automated stock trading: An ensemble strategy (2020)
  134. MAPS: Multi-Agent reinforcement learning-based Portfolio management System (2020) — link
  135. Deep graph convolutional reinforcement learning for financial portfolio management–deeppocket (2021)
  136. A survey on concept drift adaptation (2014)
  137. Deep reinforcement learning using genetic algorithm for parameter optimization (2019)
  138. Deep neuroevolution: Genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning (2017) — link
  139. Bias in natural actor-critic algorithms (2014)
  140. Q-learning (1992)
  141. A reinforcement learning method for maximizing undiscounted rewards (1993)
  142. Continuous control with deep reinforcement learning (2015) — link
  143. Stock returns, beta, variance and size: an empirical analysis (1984)
  144. Neural Network–Based Financial Volatility Forecasting: A Systematic Review (2022)
  145. A theory of Pavlovian conditioning: Variations in the effectiveness of reinforcement and non-reinforcement (1972)
  146. Dynamic treatment regimes (2014)
  147. Precision medicine (2019)

0 of 155 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.