Paper: arXiv 2512.10971

Authors: Tianyu Fan, Yuhao Yang, Yangqin Jiang, Yifei Zhang, Yuxuan Chen, Chao Huang

Abstract

Large Language Models (LLMs) have demonstrated remarkable potential as autonomous agents, approaching human-expert performance through advanced reasoning and tool orchestration. However, decision-making in fully dynamic and live environments remains highly challenging, requiring real-time information integration and adaptive responses. While existing efforts have explored live evaluation mechanisms in structured tasks, a critical gap remains in systematic benchmarking for real-world applications, particularly in finance where stringent requirements exist for live strategic responsiveness. To address this gap, we introduce AI-Trader, the first fully-automated, live, and data-uncontaminated evaluation benchmark for LLM agents in financial decision-making. AI-Trader spans three major financial markets: U.S. stocks, A-shares, and cryptocurrencies, with multiple trading granularities to simulate live financial environments. Our benchmark implements a revolutionary fully autonomous minimal information paradigm where agents receive only essential context and must independently search, verify, and synthesize live market information without human intervention. We evaluate six mainstream LLMs across three markets and multiple trading frequencies. Our analysis reveals striking findings: general intelligence does not automatically translate to effective trading capability, with most agents exhibiting poor returns and weak risk management. We demonstrate that risk control capability determines cross-market robustness, and that AI trading strategies achieve excess returns more readily in highly liquid markets than policy-driven environments. These findings expose critical limitations in current autonomous agents and provide clear directions for future improvements. The code and evaluation data are open-sourced to foster community research: https://github.com/HKUDS/AI-Trader.

Complexity vs Empirical Score

  • Math Complexity: 3.0/10
  • Empirical Rigor: 6.0/10 (capped from 8.0: transaction costs not modelled; see penalty flags)
  • Quadrant: Street Traders — practical and empirical, lighter on theory

Why this score: The paper focuses on benchmarking LLM agents in live markets with a data-uncontaminated, multi-market setup, indicating high implementation and empirical rigor. However, the core methodology relies on tool orchestration and agent frameworks rather than advanced mathematical modeling or statistical derivations.

Research Flowchart

  flowchart TD
  A["Research Goal<br>Benchmark LLM Agents in Real-Time Markets"] --> B["Methodology: AI-Trader<br>Live, Automated, Minimal Info Paradigm"]
  B --> C["Data: 3 Markets<br>US Stocks, A-shares, Crypto"]
  C --> D["Computational Process<br>Autonomous Search & Reasoning"]
  D --> E["Evaluation: 6 LLMs<br>Multi-Granularity Trading"]
  E --> F["Key Findings/Outcomes<br>1. General Intelligence ≠ Trading Skill<br>2. Risk Control Drives Robustness<br>3. Liquidity Enables Excess Returns"]

Cited References (235)

  1. Claude 3.7 Sonnet (2025)
  2. Evaluation and benchmarking of llm agents: A survey (2025) — link
  3. A review of prominent paradigms for llm-based agents: Tool use, planning (including rag), and feedback learning (2025) — link
  4. Survey on evaluation of llm-based agents (2025) — link
  5. Model Context Protocol (2025)
  6. React: Synergizing reasoning and acting in language models (2022)
  7. A New Standard for Multi-Agent Intelligence in AI Systems (2025)
  8. Bizfinbench: A business-driven real-world financial benchmark for evaluating llms (2025) — link
and 227 more
  1. When thinking fails: The pitfalls of reasoning for instruction-following in llms (2025) — link
  2. Healthbench: Evaluating large language models towards improved human health (2025) — link
  3. GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks (2025) — link
  4. Introducing claude 4 (2025)
  5. Qwen2.5-VL technical report (2025) — link
  6. Agentscope: A flexible yet robust multi-agent platform (2024) — link
  7. $tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (2024) — link
  8. Gaia: a benchmark for general ai assistants (2023)
  9. Swe-bench: Can language models resolve real-world github issues? (2023) — link
  10. Finsearchcomp: Towards a realistic, expert-level evaluation of financial search and reasoning (2025) — link
  11. Time Travel is Cheating: Going Live with DeepFund for Real-Time Fund Investment Benchmarking (2025) — link
  12. INVESTORBENCH : A Benchmark for Financial Decision-Making Tasks with LLM -based Agent (2025) — link
  13. StockBench: Can LLM Agents Trade Stocks Profitably In Real-world Markets? (2025) — link
  14. Finance Agent Benchmark: Benchmarking LLMs on Real-world Financial Research Tasks (2025) — link
  15. BizFinBench: A Business-Driven Real-World Financial Benchmark for Evaluating LLMs (2025) — link
  16. Futurex: An advanced live benchmark for llm agents in future prediction (2025) — link
  17. Livecodebench: Holistic and contamination free evaluation of large language models for code (2024) — link
  18. Browsecomp: A simple yet challenging benchmark for browsing agents (2025) — link
  19. xbench: Tracking Agents Productivity Scaling with Profession-Aligned Real-World Evaluations (2025) — link
  20. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension (2017) — link
  21. Agentorchestra: A hierarchical multi-agent framework for general-purpose task solving (2025)
  22. AutoAgent: A Fully-Automated and Zero-Code Framework for LLM Agents (2025) — link
  23. Scaling Learning Algorithms Towards AI (2007)
  24. A Fast Learning Algorithm for Deep Belief Nets (2006)
  25. Deep learning (2016)
  26. Longwriter: Unleashing 10,000+ word generation from long context llms (2024) — link
  27. DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents (2025) — link
  28. Self-supervised learning on graphs: Contrastive, generative, or predictive (2021)
  29. Deep Clustering and Visualization for End-to-End High-Dimensional Data Analysis (2022)
  30. Teaching Yourself: Graph Self-Distillation on Neighborhood for Node Classification (2022)
  31. Knowledge Distillation Improves Graph Structure Augmentation for Graph Neural Networks (2022)
  32. Beyond Homophily and Homogeneity Assumption: Relation-Based Frequency Adaptive Graph Neural Networks (2023)
  33. Homophily-Enhanced Self-Supervision for Graph Structure Learning: Insights and Directions (2023)
  34. Extracting Low-/High-Frequency Knowledge from Graph Neural Networks and Injecting it into MLPs: An Effective GNN-to-MLP Distillation Framework (2023)
  35. Graphmixup: Improving class-imbalanced node classification by reinforcement mixup and self-supervised context prediction (2023)
  36. Quantifying the Knowledge in GNNs for Reliable Distillation into MLPs (2023)
  37. Learning to Augment Graph Structure for both Homophily and Heterophily Graphs (2023)
  38. Human memory: A proposed system and its control processes (1968)
  39. ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools (2024) — link
  40. Language models are unsupervised multitask learners (2019)
  41. Foundations of machine learning (2018)
  42. Elements of information theory (1999)
  43. Deep Learning and the Information Bottleneck Principle (2015) — link
  44. I nternet-Augmented Dialogue Generation (2022) — link
  45. BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation (2023) — link
  46. Webwalker: Benchmarking llms in web traversal (2025) — link
  47. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments (2024)
  48. OSWorld-MCP: Benchmarking MCP Tool Invocation In Computer-Use Agents (2025) — link
  49. Understanding DeepResearch via Reports (2025) — link
  50. Followbench: A multi-level fine-grained constraints following benchmark for large language models (2024)
  51. Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion (2023)
  52. Ppm: Automated generation of diverse programming problems for benchmarking code generation models (2024)
  53. Artifactsbench: Bridging the visual-interactive gap in llm code generation evaluation (2025) — link
  54. Core: Benchmarking llms code reasoning capabilities through static analysis tasks (2025) — link
  55. Retrieval Augmentation Reduces Hallucination in Conversation (2021) — link
  56. Grounding Language Model with Chunking-Free In-Context Retrieval (2024) — link
  57. LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models (2023) — link
  58. Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone (2024) — link
  59. Ya RN : Efficient Context Window Extension of Large Language Models (2024) — link
  60. BGE Landmark Embedding: A Chunking-Free Embedding Method For Retrieval Augmented Long-Context Large Language Models (2024) — link
  61. Learning to Filter Context for Retrieval-Augmented Generation (2023) — link
  62. From Local to Global: A Graph RAG Approach to Query-Focused Summarization (2024) — link
  63. Qwen2 technical report (2024) — link
  64. A Human-Inspired Reading Agent with Gist Memory of Very Long Contexts (2024) — link
  65. Knowledge Graph Prompting for Multi-Document Question Answering (2024) — link
  66. RQ-RAG: Learning to Refine Queries for Retrieval Augmented Generation (2024) — link
  67. Are Long-LLMs A Necessity For Long-Context Tasks? (2024) — link
  68. Precise Zero-Shot Dense Retrieval without Relevance Labels (2022) — link
  69. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama (2023) — link
  70. GQA : Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (2023) — link
  71. Flashattention: Fast and memory-efficient exact attention with io-awareness (2022)
  72. HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models (2024) — link
  73. Retrieval-Augmented Generation for Large Language Models: A Survey (2024) — link
  74. Optimizing Factual Accuracy in Text Generation through Dynamic Knowledge Selection (2023) — link
  75. WebBrain: Learning to Generate Factually Correct Articles for Queries by Grounding on Large Web Corpus (2023) — link
  76. Large Language Models Know Your Contextual Search Intent: A Prompting Framework for Conversational Search (2023) — link
  77. Webformer: Pre-training with Web Pages for Information Retrieval (2022) — link
  78. R etrieval- A ugmented G eneration for Knowledge-Intensive NLP Tasks (2020) — link
  79. A survey on long text modeling with transformers (2023) — link
  80. Survey of hallucination in natural language generation (2023)
  81. Faithfulness in natural language generation: A systematic survey of analysis, evaluation and optimization methods (2022) — link
  82. Challenges and applications of large language models (2023) — link
  83. Distilling Knowledge from Reader to Retriever for Question Answering (2021) — link
  84. Replug: Retrieval-augmented black-box language models (2023) — link
  85. W izard of W ikipedia: Knowledge-Powered Conversational Agents (2019) — link
  86. Distilling Knowledge from Reader to Retriever for Question Answering (2021)
  87. REALM : Retrieval-Augmented Language Model Pre-Training (2020) — link
  88. R3: Reinforced ranker-reader for open-domain question answering (2018) — link
  89. Passage Re-ranking with BERT (2020) — link
  90. Large Language Models Can Be Easily Distracted by Irrelevant Context (2023) — link
  91. Generalization through memorization: Nearest neighbor language models (2019) — link
  92. EASE: Entity-aware contrastive learning of sentence embedding (2022) — link
  93. Large Language Models with Controllable Working Memory (2022) — link
  94. Knowledge Graph-Augmented Language Models for Knowledge-Grounded Dialogue Generation (2023) — link
  95. In-context retrieval-augmented language models (2023) — link
  96. Auto-GPT for Online Decision Making: Benchmarks and Additional Opinions (2023) — link
  97. Language models are few-shot learners (2020)
  98. GPT-4 Technical Report (2023)
  99. MiniMax-M2 (2025)
  100. GPT-5 (2025)
  101. Claude 3.7 sonnet and claude code. (2025)
  102. Gemini: A Family of Highly Capable Multimodal Models (2023)
  103. Llama 2: Open foundation and fine-tuned chat models (2023) — link
  104. Guided generation of cause and effect (2021)
  105. Retrieval meets Long Context Large Language Models (2023) — link
  106. Reflexion: an autonomous agent with dynamic memory and self-reflection (2023) — link
  107. Parallel Context Windows Improve In-Context Learning of Large Language Models (2022) — link
  108. A survey on in-context learning (2022) — link
  109. Palm-e: An embodied multimodal language model (2023) — link
  110. Auto-gpt for online decision making: Benchmarks and additional opinions (2023) — link
  111. The NarrativeQA Reading Comprehension Challenge (2017) — link
  112. Exploring the limits of transfer learning with a unified text-to-text transformer (2020)
  113. GPT-3.5 Turbo fine-tuning and API updates (2023) — [link](https://openai.com/blog/gpt-3-5-turbo-fine-tuning-and-\ api-updates)
  114. GPT-4 Technical Report (2023) — link
  115. Model card and evaluations for Claude models (2023) — link
  116. Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering (2021)
  117. OpenOrca: An Open Dataset of GPT Augmented FLAN Reasoning Traces (2023)
  118. Mistral 7B (2023) — link
  119. Pytorch fsdp: experiences on scaling fully sharded data parallel (2023) — link
  120. Social cognition and the human brain (1999)
  121. Computer systems: a programmer’s perspective (2011)
  122. GLM: General Language Model Pretraining with Autoregressive Blank Infilling (2022)
  123. How Long Can Context Length of Open-Source LLMs truly Promise? (2023)
  124. In-Context Pretraining: Language Modeling Beyond Document Boundaries (2023)
  125. Introducing mpt-30b: Raising the bar for open-source foundation models (2023)
  126. Together 32K (2023) — link
  127. Effective Long-Context Scaling of Foundation Models (2023) — link
  128. RWKV: Reinventing RNNs for the Transformer Era (2023)
  129. Mamba: Linear-Time Sequence Modeling with Selective State Spaces (2023) — link
  130. Random-access infinite context length for transformers (2024)
  131. Large Language Models for Information Retrieval: A Survey (2024) — link
  132. Data Engineering for Scaling Language Models to 128K Context (2024) — link
  133. Internlm2 technical report (2024) — link
  134. Longalign: A recipe for long context alignment of large language models (2024) — link
  135. LongWanjuan: Towards Systematic Measurement for Long Text Quality (2024) — link
  136. Yi: Open Foundation Models by 01.AI (2024) — link
  137. Extending context window of large language models via positional interpolation (2023) — link
  138. Hierarchical Context Merging: Better Long Context Understanding for Pre-trained LLMs (2023)
  139. Scaling Laws of RoPE-based Extrapolation (2023)
  140. YaRN: Efficient Context Window Extension of Large Language Models (2023)
  141. LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models (2023)
  142. LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens (2024) — link
  143. Focused transformer: Contrastive training for context scaling (2024)
  144. The narrativeqa reading comprehension challenge (2018)
  145. A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers (2021)
  146. Longbench: A bilingual, multitask benchmark for long context understanding (2023) — link
  147. Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps (2020)
  148. MuSiQue: Multihop Questions via Single-hop Question Composition (2022)
  149. Efficient Attentions for Long Document Summarization (2021)
  150. QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization (2021)
  151. Multi-News: A Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model (2019)
  152. Lost in the middle: How language models use long contexts (2024)
  153. Retrieval meets Long Context Large Language Models (2023)
  154. Prediction with a short memory (2018)
  155. Do Long-Range Language Models Actually Use Long-Range Context? (2021)
  156. Llama 2: Open foundation and fine-tuned chat models (2023) — link
  157. Measuring Massive Multitask Language Understanding (2020)
  158. BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions (2019)
  159. RACE: Large-scale ReAding Comprehension Dataset From Examinations (2017)
  160. C ommonsense QA : A Question Answering Challenge Targeting Commonsense Knowledge (2019) — link
  161. Think you have solved question answering? try arc, the ai2 reasoning challenge (2018) — link
  162. HellaSwag: Can a Machine Really Finish Your Sentence? (2019)
  163. Training verifiers to solve math word problems (2021) — link
  164. Measuring Mathematical Problem Solving With the MATH Dataset (2021)
  165. ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding (2023)
  166. L-eval: Instituting standardized evaluation for long context language models (2023) — link
  167. LooGLE: Can Long-Context Language Models Understand Long Contexts? (2023) — link
  168. $\infty$Bench: Extending Long Context Evaluation Beyond 100K Tokens (2024) — link
  169. Longformer: The long-document transformer (2020) — link
  170. Efficient content-based sparse attention with routing transformers (2021)
  171. Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation (2021)
  172. S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models (2024) — link
  173. Long-context LLMs Struggle with Long In-context Learning (2024) — link
  174. RULER: What’s the Real Context Size of Your Long-Context Language Models? (2024) — link
  175. In-context Learning and Induction Heads (2022) — link
  176. StarCoder: may the source be with you! (2023)
  177. Make Your LLM Fully Utilize the Context (2024) — link
  178. Learning From Mistakes Makes LLM Better Reasoner (2024) — link
  179. Rouge: A package for automatic evaluation of summaries (2004)
  180. Code llama: Open foundation models for code (2023) — link
  181. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering (2018) — link
  182. Extending Llama-3’s Context Ten-Fold Overnight (2024) — link
  183. Eq-bench: An emotional intelligence benchmark for large language models (2023) — link
  184. Cognitive Kernel-Pro: A Framework for Deep Research Agents and Agent Foundation Models Training (2025) — link
  185. Qwen3 technical report (2025) — link
  186. LiveTradeBench: Seeking Real-World Alpha with Large Language Models — → 2511.03628
  187. Gpqa: A graduate-level google-proof q&a benchmark (2024)
  188. Humanity’s last exam (2025) — link
  189. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities (2025) — link
  190. Deepseek-v3 technical report (2024) — link
  191. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model (2024) — link
  192. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality (2023) — link
  193. LongCoder: A Long-Range Pre-trained Language Model for Code Completion (2023) — link
  194. SAMS um Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization (2019) — link
  195. How Long Can Open-Source LLMs Truly Promise on Context Length? (2023) — link
  196. Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model (2019) — link
  197. Efficient Attentions for Long Document Summarization (2021) — link
  198. Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps (2020) — link
  199. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena (2023) — link
  200. C-Pack: Packaged Resources To Advance General Chinese Embedding (2023) — link
  201. Retrieve Anything To Augment Large Language Models (2023) — link
  202. Scaling Laws for Neural Language Models (2020) — link
  203. Is Retriever Merely an Approximator of Reader? (2020) — link
  204. Nearest Neighbor Machine Translation (2021) — link
  205. Generalization through Memorization: Nearest Neighbor Language Models (2020) — link
  206. When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories (2023) — link
  207. Scaling Instruction-Finetuned Language Models (2022) — link
  208. BioASQ atCLEF2023: The Eleventh Edition oftheLarge-Scale Biomedical Semantic Indexing andQuestion Answering Challenge (2023) — link
  209. Evaluating Open-Domain Question Answering in the Era of Large Language Models (2023) — link
  210. Chain of Thought Prompting Elicits Reasoning in Large Language Models (2022) — link
  211. Augmented Language Models: a Survey (2023) — link
  212. Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval (2022) — link
  213. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity (2023) — link
  214. Retrieval augmentation reduces hallucination in conversation (2021) — link
  215. Llm lies: Hallucinations are not bugs, but features as adversarial examples (2023) — link
  216. RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation (2023) — link
  217. Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language Models (2023) — link
  218. TCRA-LLM: Token Compression Retrieval Augmented Large Language Model for Inference Cost Reduction (2023) — link
  219. Unlocking Context Constraints of LLMs: Enhancing Context Efficiency of LLMs with Self-Information-Based Content Filtering (2023) — link
  220. LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression (2023) — link
  221. WebGPT: Browser-assisted question-answering with human feedback (2022) — link
  222. WebGLM: Towards An Efficient Web-Enhanced Question Answering System with Human Preferences (2023) — link
  223. Lo RA : Low-Rank Adaptation of Large Language Models (2022) — link
  224. Active Retrieval Augmented Generation (2023) — link
  225. Deep Learning for Answer Sentence Selection (2014) — link
  226. LightRAG: Simple and Fast Retrieval-Augmented Generation (2024) — link
  227. MemoRAG: Moving towards Next-Gen RAG Via Memory-Inspired Knowledge Discovery (2024) — link

1 of 235 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.