Paper: arXiv 2502.17967

Abstract

Large language models (LLMs) have demonstrated remarkable capabilities in natural language tasks, yet their performance in dynamic, real-world financial environments remains underexplored. Existing approaches are limited to historical backtesting, where trading actions cannot influence market prices and agents train only on static data. To address this limitation, we present the Agent Trading Arena, a virtual zero-sum stock market in which LLM-based agents engage in competitive multi-agent trading and directly impact price dynamics. By simulating realistic bid-ask interactions, our platform enables training in scenarios that closely mirror live markets, thereby narrowing the gap between training and evaluation. Experiments reveal that LLMs struggle with numerical reasoning when given plain-text data, often overfitting to local patterns and recent values. In contrast, chart-based visualizations significantly enhance both numerical reasoning and trading performance. Furthermore, incorporating a reflection module yields additional improvements, especially with visual inputs. Evaluations on NASDAQ and CSI datasets demonstrate the superiority of our method, particularly under high volatility. All code and data are available at https://github.com/wekjsdvnm/Agent-Trading-Arena.

Complexity vs Empirical Score

  • Math Complexity: 4.0/10
  • Empirical Rigor: 8.5/10
  • Quadrant: Street Traders — practical and empirical, lighter on theory

Why this score: The paper focuses on an experimental framework (Agent Trading Arena) with extensive empirical evaluation on real datasets (NASDAQ/CSI), providing code, and reporting performance metrics, indicating high empirical rigor. The mathematics is relatively low in complexity, primarily involving market microstructure simulation logic and basic reasoning patterns rather than advanced derivations or dense formulas.

Research Flowchart

  flowchart TD
  A["Research Goal<br>Understand & improve numerical reasoning<br>of LLM-based trading agents"] --> B["Agent Trading Arena<br>Virtual zero-sum market<br>Agents impact price dynamics"]
  B --> C["Inputs & Setup<br>Dataset: NASDAQ & CSI stocks<br>Inputs: Plain-text vs. Chart-based data<br>Agents: Base LLM & + Reflection Module"]
  C --> D["Core Process<br>Competitive Multi-Agent Trading<br>Reinforcement Learning & Simulation"]
  D --> E["Key Findings & Outcomes"]
  E --> F["1. Visual Input Superiority<br>Charts > Plain-text for numerical reasoning"]
  E --> G["2. Reflection Enhances Gains<br>Reflection module improves performance, especially with charts"]
  E --> H["3. Robustness in Volatility<br>Method excels in high-volatility markets"]

Cited References (60)

  1. ChatDev: Communicative Agents for Software Development (2024)
  2. MemoryBank: Enhancing Large Language Models with Long-Term Memory (2024)
  3. Generative Agents: Interactive Simulacra of Human Behavior (2023)
  4. Social simulacra: Creating populated prototypes for social computing systems (2022)
  5. AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors in Agents (2023) — link
  6. AgentSims: An Open-Source Sandbox for Large Language Model Evaluation (2023) — link
  7. Voyager: An Open-Ended Embodied Agent with Large Language Models (2024)
  8. Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Tasks (2024)
and 52 more
  1. Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory (2023) — link
  2. ChatDB: Augmenting LLMs with Databases as Their Symbolic Memory (2023) — link
  3. MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework (2024)
  4. When ai meets finance (stockagent): Large language model-based stock trading in simulated real-world environments — → 2407.18957
  5. TradingAgents : Multi-Agents LLM Financial Trading Framework — → 2412.20138
  6. C rypto T rade: A Reflective LLM -based Agent to Guide Zero-shot Cryptocurrency Trading (2024)
  7. TransLLaMa : LLM -based Simultaneous Translation System (2024)
  8. A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models (2024)
  9. Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarization (2023) — link
  10. LLM-ARC: Enhancing LLMs with an Automated Reasoning Critic (2024) — link
  11. LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic Provers (2023)
  12. PaLM 2 Technical Report (2023) — link
  13. GPT Can Solve Mathematical Problems Without a Calculator (2023) — link
  14. Building Cooperative Embodied Agents Modularly with Large Language Models (2024)
  15. Improving Factuality and Reasoning in Language Models through Multiagent Debate (2024)
  16. Self-Collaboration Code Generation via ChatGPT (2024)
  17. SocioDojo: Building Lifelong Analytical Agents with Real-world Text and Time Series (2024)
  18. The LLaMa 3 herd of models (2024) — link
  19. GPT-4o System Card (2024) — link
  20. GPT-3.5 model card (2023) — link
  21. Qwen2.5 Technical Report (2024) — link
  22. Qwen2 Technical Report (2024) — link
  23. GPT-4 Technical Report (2023) — link
  24. Palm 2 technical report (2023) — link
  25. Llama: Open and efficient foundation language models (2023) — link
  26. GROOT: Learning to Follow Instructions by Watching Gameplay Videos (2024)
  27. SportsMetrics: Blending Text and Numerical Data to Understand Information Fusion in LLMs (2024)
  28. Reflect, Not Reflex: Inference-Based Common Ground Improves Dialogue Response Quality (2022)
  29. Reflexion: language agents with verbal reinforcement learning (2023)
  30. A Survey of Deep Learning for Mathematical Reasoning (2023)
  31. Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models (2023)
  32. Reasoning with Language Model Prompting: A Survey (2023)
  33. Analyzing Temporal Complex Events with Large Language Models? A Benchmark towards Temporal, Long Context Understanding (2024)
  34. Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution (2024) — link
  35. Qwen Technical Report (2023) — link
  36. ChatGLM : A Family of Large Language Models from GLM-130B to GLM-4 All Tools (2024) — link
  37. DeepSeek-V3 Technical Report (2024) — link
  38. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context (2024) — link
  39. Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations? (2024)
  40. Economics arena for large language models (2024) — link
  41. ChatEval: Towards Better LLM -based Evaluators through Multi-Agent Debate (2024)
  42. How Language Model Hallucinations Can Snowball (2024)
  43. Non-linear prediction of security returns with moving average rules (1996)
  44. Mean reversion in stock prices: Evidence and implications (1988)
  45. Technical analysis and the London Stock exchange: Testing the MACD and RSI rules using the FT30 (2008)
  46. StockFormer: Learning Hybrid Trading Machines with Predictive Coding (2023)
  47. TimesNet: Temporal 2D -Variation Modeling for General Time Series Analysis (2023)
  48. A Time Series is Worth 64 Words: Long-term Forecasting with Transformers (2023)
  49. PDFormer : Propagation delay-aware dynamic long-range transformer for traffic flow prediction (2023)
  50. iTransformer: Inverted Transformers Are Effective for Time Series Forecasting (2024)
  51. StockMixer : A Simple Yet Strong MLP-Based Architecture for Stock Price Forecasting (2024)
  52. StoryLLaVA: Enhancing Visual Storytelling with Multi-Modal Large Language Models (2025)

2 of 60 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.