Paper: arXiv 2409.05144

Abstract

Alpha factor mining aims to discover investment signals from the historical financial market data, which can be used to predict asset returns and gain excess profits. Powerful deep learning methods for alpha factor mining lack interpretability, making them unacceptable in the risk-sensitive real markets. Formulaic alpha factors are preferred for their interpretability, while the search space is complex and powerful explorative methods are urged. Recently, a promising framework is proposed for generating formulaic alpha factors using deep reinforcement learning, and quickly gained research focuses from both academia and industries. This paper first argues that the originally employed policy training method, i.e., Proximal Policy Optimization (PPO), faces several important issues in the context of alpha factors mining. Herein, a novel reinforcement learning algorithm based on the well-known REINFORCE algorithm is proposed. REINFORCE employs Monte Carlo sampling to estimate the policy gradient-yielding unbiased but high variance estimates. The minimal environmental variability inherent in the underlying state transition function, which adheres to the Dirac distribution, can help alleviate this high variance issue, making REINFORCE algorithm more appropriate than PPO. A new dedicated baseline is designed to theoretically reduce the commonly suffered high variance of REINFORCE. Moreover, the information ratio is introduced as a reward shaping mechanism to encourage the generation of steady alpha factors that can better adapt to changes in market volatility. Evaluations on real assets data indicate the proposed algorithm boosts correlation with returns by 3.83%, and a stronger ability to obtain excess returns compared to the latest alpha factors mining methods, which meets the theoretical results well.

Complexity vs Empirical Score

  • Math Complexity: 8.5/10
  • Empirical Rigor: 6.0/10 (capped from 7.2: transaction costs not modelled; see penalty flags)
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper presents advanced mathematical derivations on policy gradient variance and uses sophisticated RL theory (Dirac distribution analysis, variance bounds), while providing strong empirical validation with real asset data, performance metrics (correlation boost, excess returns), and comparison to prior methods.

Research Flowchart

  flowchart TD
  A["Research Goal: Discover interpretable & steady formulaic alpha factors using Deep Reinforcement Learning"]
  B["Methodology: Variance-bounded REINFORCE Algorithm<br>Monte Carlo Sampling & Dedicated Baseline"]
  C["Key Data/Inputs: Historical Financial Market Data<br>Real Asset Returns for Evaluation"]
  D["Computational Process: Policy Generation → Reward Calculation using Information Ratio → Policy Update"]
  E["Key Findings/Outcomes: 3.83% boost in correlation with returns<br>Superior excess returns vs. latest methods<br>Steady factors adapting to market volatility"]

  A --> B
  B --> C
  C --> D
  D --> E

Cited References (85)

  1. Proximal policy optimization algorithms (2017) — link
  2. Machine learning and intelligent communications (2018)
  3. Q-learning-based spectrum access for multimedia transmission over cognitive radio networks (2020)
  4. Machine learning for communication performance enhancement (2018)
  5. High-dimensional continuous control using generalized advantage estimation (2015) — link
  6. China Index Research (2022) — link
  7. Trust region policy optimization (2015)
  8. Evolutionary reinforcement learning via cooperative coevolutionary negatively correlated search (2022)
and 77 more
  1. Parallel exploration via negatively correlated search (2021)
  2. Asynchronous Methods for Deep Reinforcement Learning (2016) — link
  3. Accurate label refinement from multiannotator of remote sensing data (2023)
  4. Multiscale Temporal Dynamic Learning for Time Series Classification (2025)
  5. 101 formulaic alphas (2016)
  6. Interpreting black-box models: a review on explainable artificial intelligence (2024)
  7. Attention is all you need (2017)
  8. Long Short-Term Memory (1997) — link
  9. Deep learning in the stock market—a systematic survey of practice, backtesting, and applications (2023)
  10. Survey on the application of deep learning in algorithmic trading (2021)
  11. Stock price forecast with deep learning (2020)
  12. Stock price prediction using CNN and LSTM-based deep learning models (2020)
  13. Stock market prediction via deep learning techniques: A survey — → 2212.12717
  14. Approximate confidence intervals (1953)
  15. Markov decision processes: discrete stochastic dynamic programming (2014)
  16. Autoalpha: an efficient hierarchical evolutionary algorithm for mining alpha factors in quantitative investment (2020) — link
  17. Simlob: Learning representations of limited order book for financial market simulation (2024) — link
  18. Alleviating Non-identifiability: a High-fidelity Calibration Objective for Financial Market Simulation with Multivariate Time Series Data — → 2407.16566
  19. Reducing idleness in financial cloud services via multi-objective evolutionary reinforcement learning based load balancer (2024)
  20. Qlib: An ai-oriented quantitative investment platform (2020) — link
  21. Actor-critic algorithms (1999)
  22. Stable-baselines3: Reliable reinforcement learning implementations (2021)
  23. OpenFE: automated feature generation with expert-level performance (2023)
  24. Monte Carlo tree search: A review of recent modifications and applications (2023)
  25. A multi-agent virtual market model for generalization in reinforcement learning based trading strategies (2023)
  26. A multi-agent deep reinforcement learning framework for algorithmic trading in financial markets (2022) — link
  27. FinRL: A deep reinforcement learning library for automated stock trading in quantitative finance (2020) — link
  28. Prediction of tapered steel plate girders shear strength using multigene genetic programming (2023)
  29. A Fast Successive QP Algorithm for General Mean-Variance Portfolio Optimization (2023) — link
  30. Efficient and Scalable Parametric High-Order Portfolios Design via the Skew-$t$ Distribution (2023) — link
  31. Dynamic Shrinkage Estimation of the High-Dimensional Minimum-Variance Portfolio (2023) — link
  32. Neural Augmented Kalman Filtering With Bollinger Bands for Pairs Trading (2024) — link
  33. Deeplob: Deep convolutional neural networks for limit order books (2019)
  34. Why do tree-based models still outperform deep learning on typical tabular data? (2022)
  35. Artificial Intelligence Stock Selection: Generalized Linear Model (2017)
  36. AlphaNet: Neural Network for Factor Mining (2020)
  37. AI-Infused algorithmic trading: genetic algorithms and machine learning in high-frequency trading (2023)
  38. Pyramid-Net: Intra-layer pyramid-scale feature aggregation network for retinal vessel segmentation (2021)
  39. An objective evaluation method for automated vehicle virtual test (2022)
  40. Prediction of small-molecule compound solubility in organic solvents by machine learning algorithms (2021)
  41. The optimal reward baseline for gradient-based reinforcement learning (2013) — link
  42. Sample efficient reinforcement learning with REINFORCE (2021)
  43. Variance Reduction Techniques for Gradient Estimates in Reinforcement Learning. (2004)
  44. Reinforcement comparison (1991)
  45. Revisiting AlphaNet: Structure and Feature Optimization (2020)
  46. Reinforcement-learning connectionist systems (1987)
  47. Simple statistical gradient-following algorithms for connectionist reinforcement learning (1992)
  48. AlphaNet Improvement: Structure and Loss Function (2021)
  49. Approximately optimal approximate reinforcement learning (2002)
  50. Leveraging non-uniformity in first-order non-convex optimization (2021)
  51. The power of default: Measuring the effect of slippage tolerance in decentralized exchanges (2024)
  52. Serial Quantization for Sparse Time Sequences (2021)
  53. Deep unfolding multi-scale regularizer network for image denoising (2023)
  54. A low computational complexity normalized subband adaptive filter algorithm employing signed regressor of input signal (2018)
  55. High-frequency trading and price discovery (2014)
  56. Improving VWAP strategies: A dynamic volume approach (2008)
  57. Quantitative equity portfolio management: modern techniques and applications (2007)
  58. The role of baselines in policy gradient optimization (2022)
  59. Remax: A simple, effective, and efficient method for aligning large language models (2023) — link
  60. From mimic to counteract: a two-stage reinforcement learning algorithm for Google research football (2024)
  61. Deep reinforcement learning for portfolio management (2020) — link
  62. gplearn: Genetic programming in python, with a scikit-learn inspired api (2015) — link
  63. Combining Reinforcement Learning and Barrier Functions for Adaptive Risk Management in Portfolio Optimization — → 2306.07013
  64. Generating synergistic formulaic alpha collections via reinforcement learning — → 2306.12964
  65. Research on multi factor stock selection model based on LightGBM and Bayesian Optimization (2022)
  66. Alphaevolve: A learning framework to discover novel alphas in quantitative investment (2021)
  67. Application research of the xgboost-svm combination model in quantitative investment strategy (2022)
  68. TD convergence: an optimization perspective (2023)
  69. Reinforcement learning: An introduction (2018)
  70. Q-learning (1992)
  71. Active portfolio management (2000)
  72. The value of active portfolio management (2004)
  73. Quantitative trading: how to build your own algorithmic trading business (2021)
  74. SAOFTRL: A Novel Adaptive Algorithmic Framework for Enhancing Online Portfolio Selection (2024)
  75. Optimal mean-reverting portfolio with leverage constraint for statistical arbitrage in finance (2019)
  76. Mean-reverting portfolio with budget constraint (2018)
  77. Learning representations by back-propagating errors (1986)

4 of 85 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.