MintEval: Do LLMs Implement the Trading Strategy You Asked For? A Behavioural-Equivalence Benchmark for Natural-Language-to-Strategy Code

Paper: arXiv 2610.03080

Authors: Siyu Wang, Yifan Wang, Yuecheng He

Abstract

Large language models are moving from producing trading signals to writing the code that executes them. The failure mode of the second role is silent: generated code runs, a backtest plots, yet the risk logic that the trader described is not the logic being executed. Existing code benchmarks test functional correctness on unit tests and finance benchmarks test forecasting; neither measures whether an implementation behaves like the strategy that was asked for. We introduce MintEval, a benchmark in which reference strategies are generated programmatically from a library of composable building blocks, back-translated into colloquial trader instructions, and re-implemented by the model under test. Generated and reference programs are executed bar by bar on identical market data and frictions, and compared on their actions rather than on code similarity or profit: alpha is differenced away. MintEval v0 contains 800 tasks on BTCUSDT 15-minute data, stratified by an execution-measured state-span complexity tau that is decoupled from description length. Low-cost models reach a mean ActionMatch of at most 0.544 and reproduce at most 0.087 of tasks exactly; on a stratified subset of 200 tasks a frontier model (Claude Opus 5.5) reaches 0.889 and reproduces 0.575 exactly, yet still fails silently on 0.275 of tasks. Given a menu of building blocks, models identify the strategy almost perfectly, yet 79.2% of the implementations whose specification was read correctly diverge on more than 10% of active bars. The LLM judge of a recent strategy-generation benchmark, applied verbatim, accepts every one of these silent failures.

Complexity vs Empirical Score

• Math Complexity: 6.0/10
• Empirical Rigor: 9.0/10
• Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: This paper presents a highly novel and rigorously designed benchmark for evaluating LLMs in generating trading strategy code. The empirical setup is thorough, focusing on behavioral equivalence rather than just code correctness, which is a critical distinction for real-world trading applications. The methodology for generating tasks and metrics is well-defined and addresses key challenges in LLM evaluation for stateful financial programs.

Research Flowchart

  flowchart TD
    A[Research Goal: Do LLMs Implement the Trading Strategy You Asked For?] --> B(Methodology: MintEval Benchmark Creation);
    B --> C{Data/Inputs: Programmatic Strategy Generation & Backtranslation};
    C --> D(Computational Process: LLM Re-implementation & Execution Comparison);
    D --> E[Outcomes: ActionMatch & Exact Reproduction Scores];

Cited References (17)

1. Evaluating Large Language Models Trained on Code (2021) — link
2. Program Synthesis with Large Language Models (2021) — link
3. SWE -bench: Can Language Models Resolve Real-World GitHub Issues? (2024)
4. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation (2023)
5. DS-1000 : A Natural and Reliable Benchmark for Data Science Code Generation (2023)
6. BigCodeBench : Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (2024) — link
7. LiveCodeBench : Holistic and Contamination Free Evaluation of Large Language Models for Code (2024) — link
8. CodeHalu : Investigating Code Hallucinations in LLMs via Execution-based Verification (2024) — link

2 of 17 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.