Paper: arXiv 2311.10801

Abstract

Portfolio management (PM) is a fundamental financial trading task, which explores the optimal periodical reallocation of capitals into different stocks to pursue long-term profits. Reinforcement learning (RL) has recently shown its potential to train profitable agents for PM through interacting with financial markets. However, existing work mostly focuses on fixed stock pools, which is inconsistent with investors’ practical demand. Specifically, the target stock pool of different investors varies dramatically due to their discrepancy on market states and individual investors may temporally adjust stocks they desire to trade (e.g., adding one popular stocks), which lead to customizable stock pools (CSPs). Existing RL methods require to retrain RL agents even with a tiny change of the stock pool, which leads to high computational cost and unstable performance. To tackle this challenge, we propose EarnMore, a rEinforcement leARNing framework with Maskable stOck REpresentation to handle PM with CSPs through one-shot training in a global stock pool (GSP). Specifically, we first introduce a mechanism to mask out the representation of the stocks outside the target pool. Second, we learn meaningful stock representations through a self-supervised masking and reconstruction process. Third, a re-weighting mechanism is designed to make the portfolio concentrate on favorable stocks and neglect the stocks outside the target pool. Through extensive experiments on 8 subset stock pools of the US stock market, we demonstrate that EarnMore significantly outperforms 14 state-of-the-art baselines in terms of 6 popular financial metrics with over 40% improvement on profit.

Complexity vs Empirical Score

  • Math Complexity: 7.5/10
  • Empirical Rigor: 8.0/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper employs advanced RL theory, deep learning architectures, and representation learning with self-supervised objectives, indicating high mathematical complexity. It also features extensive empirical validation with multiple baselines, specific financial metrics, and a public code repository, demonstrating strong empirical rigor.

Research Flowchart

  flowchart TD
  A["Research Goal: Handle Portfolio Management with Customizable Stock Pools<br/>using one-shot training"] --> B["Data Input: Global Stock Pool & Financial Markets"]
  B --> C["Methodology: EarnMore Framework"]
  C --> D["Representation Learning<br/>Self-supervised Masking & Reconstruction"]
  C --> E["Masking Mechanism<br/>Mask representations of stocks outside target pool"]
  C --> F["Re-weighting Mechanism<br/>Concentrate on favorable stocks"]
  D & E & F --> G["Computational Process<br/>RL Agent Interaction with CSPs"]
  G --> H["Key Findings<br/>Outperforms 14 SOTA baselines<br/>>40% profit improvement"]

Cited References (60)

  1. Online portfolio selection: A survey (2014)
  2. Market making via reinforcement learning (2018)
  3. Universal trading for order execution with oracle policy distillation (2021)
  4. Mastering the game of go without human knowledge (2017)
  5. An end-to-end optimal trade execution framework based on proximal policy optimization (2020)
  6. Discovering faster matrix multiplication algorithms with reinforcement learning (2022)
  7. Reinforcement learning for quantitative trading (2023)
  8. Time series momentum (2012)
and 52 more
  1. Bollinger on bollinger bands (2002)
  2. Deep direct reinforcement learning for financial signal representation and trading (2016)
  3. Hierarchical multi-scale gaussian transformer for stock movement prediction (2020)
  4. Efficient capital markets: A review of theory and empirical work (1970)
  5. Temporal relational ranking for stock prediction (2019)
  6. Lightgbm: A highly efficient gradient boosting decision tree (2017)
  7. An actor-critic-based portfolio investment method inspired by benefit-risk optimization (2018)
  8. Model-based deep reinforcement learning for dynamic portfolio optimization (2019) — link
  9. Adaptive quantitative trading: An imitative deep reinforcement learning approach (2020)
  10. Portfolio management system with reinforcement learning (2020)
  11. Adversarial deep reinforcement learning in portfolio management (2018) — link
  12. Reinforcement-learning based portfolio management with augmented asset movement prediction states (2020)
  13. Jointly modeling transfer learning of industrial chain information and deep learning for stock prediction (2022)
  14. Transfer Learning for Portfolio Optimization (2023) — link
  15. Deep Reinforcement Learning for ESG financial portfolio management — → 2307.09631
  16. Asynchronous methods for deep reinforcement learning (2016)
  17. Trust region policy optimization (2015)
  18. Proximal policy optimization algorithms (2017) — link
  19. Playing atari with deep reinforcement learning (2013) — link
  20. Soft actor-critic algorithms and applications (2018) — link
  21. Continuous control with deep reinforcement learning (2015) — link
  22. Addressing function approximation error in actor-critic methods (2018)
  23. Performance functions and reinforcement learning for trading systems and portfolios (1998)
  24. A deep reinforcement learning framework for the financial portfolio management problem (2017) — link
  25. Investor-imitator: A framework for trading knowledge extraction (2018)
  26. DeepTrader: A deep reinforcement learning approach for risk-return balanced portfolio management with market conditions Embedding (2021)
  27. DeepScalper: A risk-aware reinforcement learning framework to capture fleeting intraday trading opportunities (2022)
  28. Commission fee is not enough: A hierarchical reinforced framework for portfolio management (2021)
  29. Masked autoencoders are scalable vision learners (2022)
  30. A time series is worth 64 words: Long-term forecasting with transformers (2022) — link
  31. SimMTM: A simple pre-training framework for masked time-series modeling (2023) — link
  32. Ti-MAE: Self-supervised masked time series autoencoders (2023) — link
  33. Qlib: An AI-oriented quantitative unvestment platform (2020) — link
  34. Mage: Masked generative encoder to unify representation learning and image synthesis (2023)
  35. FinRL: A deep reinforcement learning library for automated stock trading in quantitative finance (2020) — link
  36. TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis (2023)
  37. Policy gradient methods for reinforcement learning with function approximation (1999)
  38. Mean reversion in stock prices: Evidence and implications (1988)
  39. Cross-sectional and time-series determinants of momentum returns (2002)
  40. Xgboost: A scalable tree boosting system (2016)
  41. A dual-stage attention-based recurrent neural network for time series prediction (2017) — link
  42. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling (2018) — link
  43. A unified theory of underreaction, momentum trading, and overreaction in asset markets (1999)
  44. Accurate multivariate stock movement prediction via data-axis transformer with multi-level contexts (2021)
  45. Deep reinforcement learning for portfolio management of markets with a dynamic number of assets (2021)
  46. Technical analysis of the financial markets: A comprehensive guide to trading methods and applications (1999)
  47. 101 formulaic alphas (2016)
  48. SimMIM: A Simple Framework for Masked Image Modeling (2022)
  49. Bert: Pre-training of deep bidirectional transformers for language understanding (2018) — link
  50. FinRL: A Deep Reinforcement Learning Library for Automated Stock Trading in Quantitative Finance (2022) — link
  51. Reinforcement learning in financial markets-a survey (2018)
  52. TradeMaster: A Holistic Quantitative Trading Platform Empowered by Reinforcement Learning (2023)

1 of 60 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.