Paper: arXiv 2402.16609

Abstract

As a model-free algorithm, deep reinforcement learning (DRL) agent learns and makes decisions by interacting with the environment in an unsupervised way. In recent years, DRL algorithms have been widely applied by scholars for portfolio optimization in consecutive trading periods, since the DRL agent can dynamically adapt to market changes and does not rely on the specification of the joint dynamics across the assets. However, typical DRL agents for portfolio optimization cannot learn a policy that is aware of the dynamic correlation between portfolio asset returns. Since the dynamic correlations among portfolio assets are crucial in optimizing the portfolio, the lack of such knowledge makes it difficult for the DRL agent to maximize the return per unit of risk, especially when the target market permits short selling (i.e., the US stock market). In this research, we propose a hybrid portfolio optimization model combining the DRL agent and the Black-Litterman (BL) model to enable the DRL agent to learn the dynamic correlation between the portfolio asset returns and implement an efficacious long/short strategy based on the correlation. Essentially, the DRL agent is trained to learn the policy to apply the BL model to determine the target portfolio weights. To test our DRL agent, we construct the portfolio based on all the Dow Jones Industrial Average constitute stocks. Empirical results of the experiments conducted on real-world United States stock market data demonstrate that our DRL agent significantly outperforms various comparison portfolio choice strategies and alternative DRL frameworks by at least 42% in terms of accumulated return. In terms of the return per unit of risk, our DRL agent significantly outperforms various comparative portfolio choice strategies and alternative strategies based on other machine learning frameworks.

Complexity vs Empirical Score

  • Math Complexity: 7.5/10
  • Empirical Rigor: 8.0/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper combines advanced theoretical frameworks (Transformer DRL, Black-Litterman Bayesian model, deterministic policy gradient) with empirical backtesting on real-world US equity data, demonstrating significant performance improvements over benchmarks.

Research Flowchart

  flowchart TD
  A["Research Goal: Enhance DRL for Portfolio Optimization by Integrating<br>Dynamic Correlation Knowledge for Long/Short Strategy"] --> B["Data Input: US Stock Market Data<br>Dow Jones Industrial Average Constituents"]
  B --> C["Methodology: Hybrid DRL-Black-Litterman Model"]
  C --> D["Computational Process:<br>Transformer-based DRL Agent Learns Policy<br>to Determine Portfolio Weights using BL Model"]
  D --> E["Outcome: Dynamically Optimized Long/Short Portfolio<br>Maximizing Return per Unit of Risk"]
  E --> F["Key Findings:<br>42%+ Higher Accumulated Return<br>Superior Risk-Adjusted Performance vs. Benchmarks"]

Cited References (90)

  1. Practical machine learning: Forecasting daily financial markets directions (2023) — link
  2. StockFormer: Learning Hybrid Trading Machines with Predictive Coding (2023) — link
  3. A time-varying stock portfolio selection model based on optimized PSO-BiLSTM and multi-objective mathematical programming under budget constraints (2023) — link
  4. Online Hybrid Neural Network for Stock Price Prediction: A Case Study of High-Frequency Stock Trading in the Chinese Market (2023) — link
  5. Combining deep reinforcement learning with technical analysis and trend monitoring on cryptocurrency markets (2023) — link
  6. From deterministic to stochastic: an interpretable stochastic model-free reinforcement learning framework for portfolio optimization (2022) — link
  7. Forecasting turning points in stock price by applying a novel hybrid CNN-LSTM-ResNet model fed by 2D segmented images (2022) — link
  8. Q-Learning for Continuous-Time Linear Systems: A Data-Driven Implementation of the Kleinman Algorithm (2022) — link
and 82 more
  1. On the forecasting of multivariate financial time series using hybridization of DCC-GARCH model and multivariate ANNs (2022) — link
  2. A multi-agent deep reinforcement learning framework for algorithmic trading in financial markets (2022) — link
  3. FactorVAE: A Probabilistic Dynamic Factor Model Based on Variational Autoencoder for Predicting Cross-Sectional Stock Returns (2022) — link
  4. Cardinality-constrained portfolio selection via two-timescale duplex neurodynamic optimization (2022) — link
  5. BERT’s sentiment score for portfolio optimization: a fine-tuned view in Black and Litterman model (2022) — link
  6. Are Transformers Effective for Time Series Forecasting? (2022) — link
  7. Adaptive reinforcement learning in control design for cooperating manipulator systems (2022) — link
  8. Disturbance observer-based adaptive reinforcement learning for perturbed uncertain surface vessels. (2022) — link
  9. Breaking the Curse of Dimensionality in Multiagent State Space: A Unified Agent Permutation Framework (2022) — link
  10. Online Actor-critic Reinforcement Learning Control for Uncertain Surface Vessel Systems with External Disturbances (2022) — link
  11. Recent advances in reinforcement learning in finance (2021) — link
  12. FinRL: deep reinforcement learning framework to automate trading in quantitative finance (2021) — link
  13. Cardinality-constrained portfolio selection based on collaborative neurodynamic optimization (2021) — link
  14. Decentralized Robust Portfolio Optimization Based on Cooperative-Competitive Multiagent Systems (2021) — link
  15. Adaptive fuzzy modeling of interval-valued stream data and application in cryptocurrencies prediction (2021) — link
  16. Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting (2021) — link
  17. DeepTrader: A Deep Reinforcement Learning Approach for Risk-Return Balanced Portfolio Management with Market Conditions Embedding (2021) — link
  18. Application of Features and Neural Network to Enhance the Performance of Deep Reinforcement Learning in Portfolio Management (2021) — link
  19. A Deep Residual Shrinkage Neural Network-based Deep Reinforcement Learning Strategy in Financial Portfolio Management (2021) — link
  20. The Use of Features to Enhance the Capability of Deep Reinforcement Learning for Investment Portfolio Management (2021) — link
  21. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020) — link
  22. A deep Q-learning portfolio management framework for the cryptocurrency market (2020) — link
  23. Reinforcement-Learning based Portfolio Management with Augmented Asset Movement Prediction States (2020) — link
  24. Minimax and Biobjective Portfolio Selection Based on Collaborative Neurodynamic Optimization (2020) — link
  25. A Multi-Scale Temporal Feature Aggregation Convolutional Neural Network for Portfolio Management (2019) — link
  26. A Deep Reinforcement Learning Approach for Automated Cryptocurrency Trading (2019) — link
  27. Adaptive Input Representations for Neural Language Modeling (2018) — link
  28. Addressing Function Approximation Error in Actor-Critic Methods (2018) — link
  29. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor (2018) — link
  30. Performance evaluation of series and parallel strategies for financial time series forecasting (2017) — link
  31. Performance evaluation of series and parallel strategies for financial time series forecasting (2017) — link
  32. Distributional Reinforcement Learning with Quantile Regression (2017) — link
  33. Proximal Policy Optimization Algorithms (2017) — link
  34. A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem (2017) — link
  35. Attention is All you Need (2017) — link
  36. Layer Normalization (2016) — link
  37. Asynchronous Methods for Deep Reinforcement Learning (2016) — link
  38. Dueling Network Architectures for Deep Reinforcement Learning (2015) — link
  39. Continuous control with deep reinforcement learning (2015) — link
  40. Can a corporate network and news sentiment improve portfolio optimization using the Black–Litterman model? (2015) — link
  41. A Black–Litterman asset allocation model under Elliptical distributions (2015) — link
  42. Learning a Deep Convolutional Network for Image Super-Resolution (2014) — link
  43. Playing Atari with Deep Reinforcement Learning (2013) — link
  44. Weighted Moving Average Passive Aggressive Algorithm for Online Portfolio Selection (2013) — link
  45. On-Line Portfolio Selection with Moving Average Reversion (2012) — link
  46. CORN: Correlation-driven nonparametric learning approach for portfolio selection (2011) — link
  47. Quantitative Equity Investing: Techniques and Strategies (2010)
  48. Optimal Portfolio Choice with Parameter Uncertainty (2007) — link
  49. Algorithms for portfolio management based on the Newton method (2006) — link
  50. NONPARAMETRIC KERNEL‐BASED SEQUENTIAL INVESTMENT STRATEGIES (2006) — link
  51. The Markowitz Optimization Enigma: Is ‘Optimized’ Optimal? (2005) — link
  52. Can We Learn to Beat the Best Stock (2003) — link
  53. Value-at-Risk for long and short trading positions (2003) — link
  54. Modern Investment Management: An Equilibrium Approach (2003)
  55. The Intuition Behind Black-Litterman Model Portfolios (2002) — link
  56. The Sveriges Riksbank (Bank of Sweden) Prize in Economic Sciences in Memory of Alfred Nobel 1969–2000 (2001) — link
  57. On‐Line Portfolio Selection Using Multiplicative Updates (1998) — link
  58. Reinforcement Learning: A Survey (1996) — link
  59. Universal portfolios with side information (1996) — link
  60. The Influence of the Sigmoid Function Parameters on the Speed of Backpropagation Learning (1995) — link
  61. Global Portfolio Optimization (1992) — link
  62. Bayes-Stein Estimation for Portfolio Analysis (1986) — link
  63. Portfolio Selection (1971) — link
  64. GPM: A graph convolutional network based reinforcement learning framework for portfolio management (2022) — link
  65. A Framework of Hierarchical Deep Q-Network for Portfolio Management (2021) — link
  66. Advances in financial machine learning (2018)
  67. Online portfolio selection: A survey (2012) — link
  68. Sortino: a ‘sharper’ratio (2013)
  69. Efficient backdrop (2012)
  70. Efficient asset management: a practical guide to stock portfolio optimization and asset allocation (2008)
  71. A step-by-step guide to the Black-Litterman model: Incorporating user-specified confidence levels (2007)
  72. The adaptive markets hypothesis: Market effi-ciency from an evolutionary perspective (2004)
  73. Publisher’s Note (2003) — link
  74. Markowitz Revisited: Mean-Variance Models in Financial Portfolio Analysis (2001) — link
  75. Theory and methodology of tactical asset allocation (Vol. 65) (2000)
  76. Universal Portfolios (1996) — link
  77. Short Selling and Common Stock Prices (1994) — link
  78. Asset allocation: combining investor views with market equilibrium (1990)
  79. Bayesian decision problemas and Markov chains (No. 519.233 M3) (1967)
  80. Machine Learning manuscript No. (will be inserted by the editor) PAMR: Passive Aggressive Mean Reversion Strategy for Portfolio Selection
  81. (2022) Adaptive optimal tracking control of an underactuated surface vessel using actor–critic reinforcement learning
  82. del Este, Uruguay, April 10–14, 2000

0 of 90 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.