Paper: arXiv 2407.14486

Abstract

Financial portfolio management investment policies computed quantitatively by modern portfolio theory techniques like the Markowitz model rely on a set on assumptions that are not supported by data in high volatility markets. Hence, quantitative researchers are looking for alternative models to tackle this problem. Concretely, portfolio management is a problem that has been successfully addressed recently by Deep Reinforcement Learning (DRL) approaches. In particular, DRL algorithms train an agent by estimating the distribution of the expected reward of every action performed by an agent given any financial state in a simulator. However, these methods rely on Deep Neural Networks model to represent such a distribution, that although they are universal approximator models, they cannot explain its behaviour, given by a set of parameters that are not interpretable. Critically, financial investors policies require predictions to be interpretable, so DRL agents are not suited to follow a particular policy or explain their actions. In this work, we developed a novel Explainable Deep Reinforcement Learning (XDRL) approach for portfolio management, integrating the Proximal Policy Optimization (PPO) with the model agnostic explainable techniques of feature importance, SHAP and LIME to enhance transparency in prediction time. By executing our methodology, we can interpret in prediction time the actions of the agent to assess whether they follow the requisites of an investment policy or to assess the risk of following the agent suggestions. To the best of our knowledge, our proposed approach is the first explainable post hoc portfolio management financial policy of a DRL agent. We empirically illustrate our methodology by successfully identifying key features influencing investment decisions, which demonstrate the ability to explain the agent actions in prediction time.

Complexity vs Empirical Score

  • Math Complexity: 6.5/10
  • Empirical Rigor: 3.5/10
  • Quadrant: Lab Rats — theoretically deep, empirically untested

Why this score: The paper employs advanced deep reinforcement learning (PPO) and XAI techniques (SHAP, LIME), requiring significant mathematical complexity. However, the empirical validation appears theoretical and lacks concrete backtesting metrics, code, or implementation details, placing it in the theoretical research quadrant.

Research Flowchart

  flowchart TD
  A["Research Goal:<br>Explainable DRL Portfolio Management"] --> B["Data Input:<br>Financial Market Simulator"]
  B --> C["Core Methodology:<br>Explainable XDRL<br>PPO + SHAP/LIME"]
  C --> D["Computational Process:<br>Training Agent &<br>Generating Explanations"]
  D --> E["Key Outcome 1:<br>Identified Key Features<br>for Investment Decisions"]
  D --> F["Key Outcome 2:<br>Post-hoc Interpretability<br>of DRL Actions"]
  E --> G["Conclusion:<br>Validated Transparent &<br>Explainable Financial Policy"]
  F --> G

Cited References (59)

  1. Proximal policy optimization algorithms (2017) — link
  2. Principles and practice of explainable machine learning (2021)
  3. The shapley value in machine learning (2022) — link
  4. Statistical stability indices for LIME: Obtaining reliable explanations for machine learning models (2022)
  5. Explainable artificial intelligence: an analytical review (2021)
  6. Intelligible and explainable machine learning: Best practices and practical challenges (2020)
  7. Deep reinforcement learning: A brief survey (2017)
  8. Reinforcement learning: An introduction (2018)
and 51 more
  1. Reinforcement learning: An introduction, 2nd ed. (2020)
  2. Openai gym (2016) — link
  3. Deep learning (2015)
  4. A universal approximation theorem of deep neural networks for expressing probability distributions (2020)
  5. Can we open the black box of AI ? (2016)
  6. Financial transparency and corporate governance: you manage what you measure (1996)
  7. The basics of finance: An introduction to financial markets, business finance, and portfolio management (2010)
  8. A simplified perspective of the Markowitz portfolio theory (2013)
  9. True Markowitz or assumptions we break and why it matters (2012)
  10. Stock market prediction and Portfolio selection models: a survey (2017)
  11. The different faces of volatility exposure in portfolio management (2013)
  12. Recent advances in reinforcement learning in finance (2023)
  13. Human-level control through deep reinforcement learning (2015)
  14. Policy gradient methods for reinforcement learning with function approximation (1999)
  15. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor (2018)
  16. Twin-delayed ddpg: A deep reinforcement learning technique to model a continuous movement of an intelligent robot agent (2019)
  17. A deep reinforcement learning framework for the financial portfolio management problem (2017) — link
  18. Cryptocurrency portfolio management with deep reinforcement learning (2017)
  19. Continuous control with deep reinforcement learning (2015) — link
  20. Deep reinforcement learning for the control of robotic manipulation: a focussed mini-review (2021)
  21. A survey of deep reinforcement learning in video games (2019) — link
  22. Explainable deep reinforcement learning: state of the art and challenges (2022)
  23. Adversarial deep reinforcement learning in portfolio management (2018) — link
  24. Deep reinforcement learning in cryptocurrency market making (2019) — link
  25. Recommending cryptocurrency trading points with deep reinforcement learning approach (2020)
  26. Deep reinforcement learning for trading (2019) — link
  27. Deep hedging: hedging derivatives under generic market frictions using reinforcement learning (2019)
  28. Deep hedging of derivatives using reinforcement learning (2021) — link
  29. Deep hedging of long-term financial derivatives (2021)
  30. Multi-agent reinforcement learning approach for hedging portfolio problem (2021)
  31. Modelling stock markets by multi-agent reinforcement learning (2021)
  32. MSPM: A modularized and scalable multi-agent reinforcement learning-based system for financial portfolio management (2022)
  33. MAPS: Multi-agent reinforcement learning-based portfolio management system (2020) — link
  34. Cost-sensitive portfolio selection via deep reinforcement learning (2020)
  35. GPM: A graph convolutional network based reinforcement learning framework for portfolio management (2022)
  36. Stock portfolio management by using fuzzy ensemble deep reinforcement learning algorithm (2023)
  37. Deep reinforcement learning for stock portfolio optimization by connecting with modern portfolio theory (2023)
  38. Market sentiment-aware deep reinforcement learning approach for stock portfolio allocation (2021)
  39. Optimistic bull or pessimistic bear: Adaptive deep reinforcement learning for stock portfolio allocation (2019) — link
  40. Leveraging Deep Learning and Online Source Sentiment for Financial Portfolio Management — → 2309.16679
  41. Stock trading bot using deep reinforcement learning (2019)
  42. FinRL: Deep reinforcement learning framework to automate trading in quantitative finance (2021)
  43. FinRL: A deep reinforcement learning library for automated stock trading in quantitative finance (2020) — link
  44. Practical deep reinforcement learning approach for stock trading (2018) — link
  45. Explainable deep reinforcement learning for portfolio management: an empirical approach (2021)
  46. Towards interpretable reinforcement learning with state abstraction driven by external knowledge (2020)
  47. X PM : An explainable deep reinforcement learning framework for portfolio management (2021)
  48. Alphastock: A buying-winners-and-selling-losers investment strategy using interpretable deep reinforcement attention networks (2019)
  49. Explainable AI: Foundations, methodologies and applications (2023)
  50. A unified approach to interpreting model predictions (2017)
  51. " W hy should I trust you?" Explaining the predictions of any classifier (2016)

1 of 59 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.