Paper: arXiv 2507.20263

Authors: Junjie Zhao, Chengxi Zhang, Chenkai Wang, Peng Yang

Abstract

Reinforcement learning (RL) has successfully automated the complex process of mining formulaic alpha factors, for creating interpretable and profitable investment strategies. However, existing methods are hampered by the sparse rewards given the underlying Markov Decision Process. This inefficiency limits the exploration of the vast symbolic search space and destabilizes the training process. To address this, Trajectory-level Reward Shaping (TLRS), a novel reward shaping method, is proposed. TLRS provides dense, intermediate rewards by measuring the subsequence-level similarity between partially generated expressions and a set of expert-designed formulas. Furthermore, a reward centering mechanism is introduced to reduce training variance. Extensive experiments on six major Chinese and U.S. stock indices show that TLRS significantly improves the predictive power of mined factors, boosting the Rank Information Coefficient by 9.29% over existing potential-based shaping algorithms. Notably, TLRS achieves a major leap in computational efficiency by reducing its time complexity with respect to the feature dimension from linear to constant, which is a significant improvement over distance-based baselines.

Complexity vs Empirical Score

  • Math Complexity: 8.0/10
  • Empirical Rigor: 7.5/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper presents advanced mathematics, including MDPs, reward shaping theory, and complexity analysis, while also demonstrating strong empirical validation with extensive experiments on six stock indices and significant performance metrics.

Research Flowchart

  flowchart TD
  A["Research Goal<br>Address sparse rewards in<br>Formulaic Alpha Mining via RL"] --> B{"Methodology: Trajectory-level<br>Reward Shaping TLRS"}
  B --> C["Input: Expert Formulas<br>& Symbolic Search Space"]
  C --> D["Compute Subsequence<br>Similarity Dense Rewards"]
  D --> E["Reward Centering<br>Reduce Training Variance"]
  E --> F["RL Agent Optimization<br>with TLRS"]
  F --> G["Outcomes: 6 Major Stock Indices"]
  G --> H["Key Findings<br>9.29% Rank IC Increase<br>Constant Time Complexity"]

Cited References (91)

  1. Accurate label refinement from multiannotator of remote sensing data (2023)
  2. Multiscale Temporal Dynamic Learning for Time Series Classification (2025)
  3. Machine learning for communication performance enhancement (2018)
  4. Machine learning and intelligent communications (2018)
  5. Q-learning-based spectrum access for multimedia transmission over cognitive radio networks (2020)
  6. Proximal policy optimization algorithms (2017) — link
  7. Learning from demonstration (1996)
  8. Maximum entropy inverse reinforcement learning. (2008)
and 83 more
  1. Principled methods for advising reinforcement learning agents (2003)
  2. Policy invariance under reward transformations: Theory and application to reward shaping (1999)
  3. A cold-start-free reinforcement learning approach for traffic signal control (2022)
  4. Learning continuous semantic representations of symbolic expressions (2017)
  5. Multi-resolution exploration in continuous spaces (2008)
  6. Apprenticeship learning via inverse reinforcement learning (2004)
  7. A survey of robot learning from demonstration (2009)
  8. Reinforcement learning from demonstration through shaping (2015)
  9. Learning to Drive a Bicycle Using Reinforcement Learning and Shaping. (1998)
  10. Robot shaping: Developing autonomous agents through learning (1994)
  11. High-dimensional continuous control using generalized advantage estimation (2015) — link
  12. China Index Research (2022) — link
  13. Trust region policy optimization (2015)
  14. Evolutionary reinforcement learning via cooperative coevolutionary negatively correlated search (2022)
  15. Parallel exploration via negatively correlated search (2021)
  16. Asynchronous Methods for Deep Reinforcement Learning (2016) — link
  17. Learning representations by back-propagating errors (1986)
  18. 101 formulaic alphas (2016)
  19. Interpreting black-box models: a review on explainable artificial intelligence (2024)
  20. Attention is all you need (2017)
  21. Long Short-Term Memory (1997) — link
  22. Deep learning in the stock market—a systematic survey of practice, backtesting, and applications (2023)
  23. Survey on the application of deep learning in algorithmic trading (2021)
  24. Stock price forecast with deep learning (2020)
  25. Stock price prediction using CNN and LSTM-based deep learning models (2020)
  26. Stock market prediction via deep learning techniques: A survey — → 2212.12717
  27. Approximate confidence intervals (1953)
  28. Markov decision processes: discrete stochastic dynamic programming (2014)
  29. Autoalpha: an efficient hierarchical evolutionary algorithm for mining alpha factors in quantitative investment (2020) — link
  30. Qlib: An ai-oriented quantitative investment platform (2020) — link
  31. Actor-critic algorithms (1999)
  32. Stable-baselines3: Reliable reinforcement learning implementations (2021)
  33. OpenFE: automated feature generation with expert-level performance (2023)
  34. Monte Carlo tree search: A review of recent modifications and applications (2023)
  35. A multi-agent virtual market model for generalization in reinforcement learning based trading strategies (2023)
  36. A multi-agent deep reinforcement learning framework for algorithmic trading in financial markets (2022) — link
  37. FinRL: A deep reinforcement learning library for automated stock trading in quantitative finance (2020) — link
  38. Prediction of tapered steel plate girders shear strength using multigene genetic programming (2023)
  39. Serial Quantization for Sparse Time Sequences (2021)
  40. Deep unfolding multi-scale regularizer network for image denoising (2023)
  41. A low computational complexity normalized subband adaptive filter algorithm employing signed regressor of input signal (2018)
  42. SAOFTRL: A Novel Adaptive Algorithmic Framework for Enhancing Online Portfolio Selection (2024)
  43. Optimal mean-reverting portfolio with leverage constraint for statistical arbitrage in finance (2019)
  44. Mean-reverting portfolio with budget constraint (2018)
  45. QuantFactor REINFORCE: Mining Steady Formulaic Alpha Factors With Variance-Bounded REINFORCE (2025) — link
  46. A Fast Successive QP Algorithm for General Mean-Variance Portfolio Optimization (2023) — link
  47. Efficient and Scalable Parametric High-Order Portfolios Design via the Skew-$t$ Distribution (2023) — link
  48. Dynamic Shrinkage Estimation of the High-Dimensional Minimum-Variance Portfolio (2023) — link
  49. Neural Augmented Kalman Filtering With Bollinger Bands for Pairs Trading (2024) — link
  50. Deeplob: Deep convolutional neural networks for limit order books (2019)
  51. Why do tree-based models still outperform deep learning on typical tabular data? (2022)
  52. Artificial Intelligence Stock Selection: Generalized Linear Model (2017)
  53. AlphaNet: Neural Network for Factor Mining (2020)
  54. AI-Infused algorithmic trading: genetic algorithms and machine learning in high-frequency trading (2023)
  55. Pyramid-Net: Intra-layer pyramid-scale feature aggregation network for retinal vessel segmentation (2021)
  56. An objective evaluation method for automated vehicle virtual test (2022)
  57. Prediction of small-molecule compound solubility in organic solvents by machine learning algorithms (2021)
  58. The optimal reward baseline for gradient-based reinforcement learning (2013) — link
  59. Sample efficient reinforcement learning with REINFORCE (2021)
  60. Variance Reduction Techniques for Gradient Estimates in Reinforcement Learning. (2004)
  61. Reinforcement comparison (1991)
  62. Revisiting AlphaNet: Structure and Feature Optimization (2020)
  63. Reinforcement-learning connectionist systems (1987)
  64. Simple statistical gradient-following algorithms for connectionist reinforcement learning (1992)
  65. AlphaNet Improvement: Structure and Loss Function (2021)
  66. Approximately optimal approximate reinforcement learning (2002)
  67. Leveraging non-uniformity in first-order non-convex optimization (2021)
  68. Quantitative equity portfolio management: modern techniques and applications (2007)
  69. The role of baselines in policy gradient optimization (2022)
  70. Remax: A simple, effective, and efficient method for aligning large language models (2023) — link
  71. The arcade learning environment: An evaluation platform for general agents (2013)
  72. From mimic to counteract: a two-stage reinforcement learning algorithm for Google research football (2024)
  73. Deep reinforcement learning for portfolio management (2020) — link
  74. gplearn: Genetic programming in python, with a scikit-learn inspired api (2015) — link
  75. Combining Reinforcement Learning and Barrier Functions for Adaptive Risk Management in Portfolio Optimization — → 2306.07013
  76. Generating synergistic formulaic alpha collections via reinforcement learning — → 2306.12964
  77. Research on multi factor stock selection model based on LightGBM and Bayesian Optimization (2022)
  78. Alphaevolve: A learning framework to discover novel alphas in quantitative investment (2021)
  79. Application research of the xgboost-svm combination model in quantitative investment strategy (2022)
  80. TD convergence: an optimization perspective (2023)
  81. Reinforcement learning: An introduction (2018)
  82. Q-learning (1992)
  83. Reward centering (2024) — link

3 of 91 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.