Execution Realism and Reproducibility in LLM-Based Trading Systems: A Systematic Scoping Review and Evidence Audit

Paper: arXiv 2606.08285

Authors: Junyi Yao, Zihao Zheng, Baichuan Li, Jiayu Long

Abstract

Execution realism remains weakly standardized in research on large-language-model-based trading systems, limiting comparison and reproduction across backtests, simulations, and portfolio benchmarks. This article presents a PRISMA-ScR systematic scoping review with a nested execution-reproducibility evidence audit. Eight reproducible query families returned 796 query-level records; DOI/title deduplication left 687 unique records, 101 reports were sought for retrieval, and 59 full texts were recovered through direct and fallback open-access routes. Fifty-three reports are provisionally eligible for expanded evidence charting, subject to reconciliation of two blind author-validation packets. The charting framework separately evaluates point-in-time controls, temporal splits, held-out evaluation, trading costs and turnover, execution semantics, universe construction, architecture reporting, and artifact availability. Direct-trading, portfolio or alpha-construction, and benchmark studies are retained as distinct descriptive subgroups rather than pooled as a common performance estimand. Search responses, retrieval attempts, file hashes, evidence excerpts, screening decisions, and coding worksheets are archived with the review materials. Final field-level counts are intentionally withheld until author validation is complete.

Complexity vs Empirical Score

• Math Complexity: 3.0/10
• Empirical Rigor: 8.0/10
• Quadrant: Street Traders — practical and empirical, lighter on theory

Why this score: This paper offers a highly rigorous and novel audit of LLM-based trading systems, focusing on critical issues of execution realism and reproducibility. While the mathematical complexity is low, its systematic approach and clear methodology make it a valuable contribution to the field.

Research Flowchart

  flowchart TD
    A[Research Goal: Assess Execution Realism & Reproducibility in LLM-Based Trading Systems] --> B{Methodology: PRISMA-ScR Systematic Scoping Review & Evidence Audit};
    B --> C[Data/Inputs: 796 Query Records (DOI/Title Deduplicated to 687 Unique Records)];
    C --> D{Computational Processes: Screening, Retrieval (101 Sought, 59 Recovered), Charting Framework Application};
    D --> E[Charting Framework: Point-in-Time Controls, Temporal Splits, Held-out Evaluation, Trading Costs/Turnover, Execution Semantics, Universe Construction, Architecture Reporting, Artifact Availability];
    E --> F[Outcomes: 53 Provisionally Eligible Reports for Expanded Charting, Distinct Subgroups (Direct-Trading, Portfolio/Alpha-Construction, Benchmark Studies), Archived Review Materials];

Cited References (119)

1. A Multi-agent System Based On LLM For Trading Financial Assets (2025) — link
2. A Reflective LLM-based Agent to Guide Zero-shot Cryptocurrency Trading — → 2407.09546
3. Agentic AI for Clustering, Relationship Discovery, and Semantic Trading in Prediction Markets (2025) — link
4. Agentic AI in Commodity Trading: A Comparative Simulation Study (2025) — link
5. AgenticAITA: A Proof-Of-Concept About Deliberative Multi-Agent Reasoning for Autonomous Trading Systems (2026) — link
6. An End-To-End LLM Enhanced Trading System — → 2502.01574
7. Analysis of the Application of Large Language Models in Three Major Fields: Trading and Portfolio Management, Financial Risk Management, and Financial Text Mining (2026) — link
8. CN-Buzz2Portfolio: A Chinese-Market Dataset and Benchmark for LLM-Based Macro and Sector Asset Allocation from Daily Trending Financial News (2026) — link

38 of 119 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.