Paper: arXiv 2508.13174

Authors: Hongjun Ding, Binqi Chen, Jinsheng Huang, Taian Guo, Zhengyang Mao, Guoyi Shao, Lutong Zou, Luchen Liu, Ming Zhang

Abstract

Formula alpha mining, which generates predictive signals from financial data, is critical for quantitative investment. Although various algorithmic approaches-such as genetic programming, reinforcement learning, and large language models-have significantly expanded the capacity for alpha discovery, systematic evaluation remains a key challenge. Existing evaluation metrics predominantly include backtesting and correlation-based measures. Backtesting is computationally intensive, inherently sequential, and sensitive to specific strategy parameters. Correlation-based metrics, though efficient, assess only predictive ability and overlook other crucial properties such as temporal stability, robustness, diversity, and interpretability. Additionally, the closed-source nature of most existing alpha mining models hinders reproducibility and slows progress in this field. To address these issues, we propose AlphaEval, a unified, parallelizable, and backtest-free evaluation framework for automated alpha mining models. AlphaEval assesses the overall quality of generated alphas along five complementary dimensions: predictive power, stability, robustness to market perturbations, financial logic, and diversity. Extensive experiments across representative alpha mining algorithms demonstrate that AlphaEval achieves evaluation consistency comparable to comprehensive backtesting, while providing more comprehensive insights and higher efficiency. Furthermore, AlphaEval effectively identifies superior alphas compared to traditional single-metric screening approaches. All implementations and evaluation tools are open-sourced to promote reproducibility and community engagement.

Complexity vs Empirical Score

  • Math Complexity: 4.0/10
  • Empirical Rigor: 6.0/10 (capped from 8.5: transaction costs not modelled; see penalty flags)
  • Quadrant: Street Traders — practical and empirical, lighter on theory

Why this score: The paper presents a pragmatic framework for alpha evaluation with modest mathematical complexity, but demonstrates high empirical rigor through open-source code, large-scale benchmarks, and specific performance metrics against traditional backtesting.

Research Flowchart

  flowchart TD
  A["Research Goal: Create AlphaEval<br>to address evaluation challenges in alpha mining"] --> B["Key Methodology: 5-Dimensional<br>Backtest-Free Evaluation Framework"]
  B --> C["Computational Process:<br>Parallelizable Assessment Pipeline"]
  C --> D["Data/Input: Representative Alpha Mining<br>Algorithms & Diverse Datasets"]
  D --> E{"Key Findings & Outcomes"}
  E --> F["✅ Evaluation Consistency<br>~ Backtesting"]
  E --> G["✅ Higher Efficiency &<br>Comprehensive Insights"]
  E --> H["✅ Superior Alpha Identification<br>vs. Single-Metric Screening"]

Cited References (55)

  1. LightGBM : A Highly Efficient Gradient Boosting Decision Tree (2017) — link
  2. XGBoost : A Scalable Tree Boosting System (2016) — link
  3. AlphaEvolve : A Learning Framework to Discover Novel Alphas in Quantitative Investment (2021) — link
  4. AutoAlpha : an Efficient Hierarchical Evolutionary Algorithm for Mining Alpha Factors in Quantitative Investment (2020) — link
  5. Alphaforge: A framework to mine and dynamically combine formulaic alpha factors (2025)
  6. Can Large Language Models Mine Interpretable Financial Factors More Effectively? A Neural-Symbolic Factor Mining Agent Model (2024) — link
  7. AlphaQCM: Alpha Discovery in Finance with Distributional Reinforcement Learning (2025)
  8. A five-factor asset pricing model (2015)
and 47 more
  1. An empirical study of financial factor mining based on gene expression programming (2021)
  2. 101 formulaic alphas (2016)
  3. Qlib: An AI-oriented Quantitative Investment Platform (2020) — link
  4. $ \textbackslash text\ Alpha \ \textasciicircum 2$: Discovering Logical Formulaic Alphas using Deep Reinforcement Learning (2024) — link
  5. RiskMiner : Discovering Formulaic Alphas via Risk Seeking Monte Carlo Tree Search (2024) — link
  6. QuantFactor REINFORCE : Mining Steady Formulaic Alpha Factors with Variance -bounded REINFORCE (2024) — link
  7. Beyond the Black Box: Interpretability of LLMs in Finance (2025) — link
  8. Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead (2019) — link
  9. Explainable Artificial Intelligence (XAI) approaches for transparency and accountability in financial decision-making (2023)
  10. Explainable AI (XAI) for Credit Scoring and Loan Approvals (2025)
  11. Enhancing explainability in real-world scenarios: towards a robust stability measure for local interpretability (2025) — link
  12. Can i trust the explanations? investigating explainable machine learning methods for monotonic models — → 2309.13246
  13. Cumulated gain-based evaluation of IR techniques (2002)
  14. A Theoretical Analysis of NDCG Type Ranking Measures (2013) — link
  15. QuantBench: Benchmarking AI Methods for Quantitative Investment (2025) — link
  16. Learning gflownets from partial episodes for improved convergence and stability (2023)
  17. A theory of continuous generative flow networks (2023)
  18. Let the flows tell: Solving graph combinatorial problems with gflownets (2023)
  19. Biological sequence design with gflownets (2022)
  20. Markov chain monte carlo and variational inference: Bridging the gap (2015)
  21. A tutorial on energy-based learning (2006)
  22. Open graph benchmark: Datasets for machine learning on graphs (2020)
  23. DistDGL: Distributed graph neural network training for billion-scale graphs (2020)
  24. Cluster-gcn: An efficient algorithm for training deep and large graph convolutional networks (2019)
  25. Distance encoding: Design provably more powerful neural networks for graph representation learning (2020)
  26. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges (2021) — link
  27. Neural message passing for quantum chemistry (2017)
  28. How Powerful are Graph Neural Networks?
  29. Graph attention networks (2017) — link
  30. Inductive representation learning on large graphs (2017)
  31. Multilayer perceptrons for classification and regression (1991)
  32. Graph convolutional networks for text classification (2019)
  33. The graph neural network model (2008)
  34. Flow network based generative models for non-iterative diverse candidate generation (2021)
  35. Trajectory balance: Improved credit assignment in gflownets (2022)
  36. Gflownet foundations (2023)
  37. Modeling relational data with graph convolutional networks (2018)
  38. Learning from Expert Factors: Trajectory-level Reward Shaping for Formulaic Alpha Mining (2025) — link
  39. 101 Formulaic Alphas (2016) — link
  40. R&D-Agent-Quant: A Multi-Agent Framework for Data-Centric Factors and Model Joint Optimization (2025)
  41. AlphaEval: A Comprehensive and Efficient Evaluation Framework for Formula Alpha Mining — → 2508.13174
  42. Navigating the Alpha Jungle: An LLM-Powered MCTS Framework for Formulaic Factor Mining (2025) — link
  43. Alphaagent: Llm-driven alpha mining with regularized exploration to counteract alpha decay (2025)
  44. Chain-of-Alpha: Unleashing the Power of Large Language Models for Alpha Mining in Quantitative Trading (2025) — link
  45. Generating Synergistic Formulaic Alpha Collections via Reinforcement Learning — → 2306.12964
  46. An empirical study of financial factor mining based on gene expression programming (2021)
  47. From Linear to Hierarchical : Evolving Tree -structured Thoughts for Efficient Alpha Mining (2025) — link

3 of 55 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.