Paper: arXiv 2412.18174

Abstract

Recent advancements have underscored the potential of large language model (LLM)-based agents in financial decision-making. Despite this progress, the field currently encounters two main challenges: (1) the lack of a comprehensive LLM agent framework adaptable to a variety of financial tasks, and (2) the absence of standardized benchmarks and consistent datasets for assessing agent performance. To tackle these issues, we introduce \textsc{“InvestorBench”}, the first benchmark specifically designed for evaluating LLM-based agents in diverse financial decision-making contexts. InvestorBench enhances the versatility of LLM-enabled agents by providing a comprehensive suite of tasks applicable to different financial products, including single equities like stocks, cryptocurrencies and exchange-traded funds (ETFs). Additionally, we assess the reasoning and decision-making capabilities of our agent framework using thirteen different LLMs as backbone models, across various market environments and tasks. Furthermore, we have curated a diverse collection of open-source, multi-modal datasets and developed a comprehensive suite of environments for financial decision-making. This establishes a highly accessible platform for evaluating financial agents’ performance across various scenarios.

Complexity vs Empirical Score

  • Math Complexity: 2.5/10
  • Empirical Rigor: 6.0/10 (capped from 8.0: transaction costs not modelled; see penalty flags)
  • Quadrant: Street Traders — practical and empirical, lighter on theory

Why this score: The paper focuses on constructing a practical benchmark (InvestorBench) with empirical evaluation using real datasets and multiple LLMs, showing strong empirical rigor, while the mathematical complexity is low as it primarily describes agent architecture without dense theoretical derivations.

Research Flowchart

  flowchart TD
  A["Research Goal<br>Create comprehensive benchmark for LLM agents<br>in diverse financial decision-making tasks"] --> B["Methodology<br>Develop InvestorBench: Multi-asset benchmark<br>with standardized environments"]
  B --> C["Data & Inputs<br>Curated open-source multi-modal datasets<br>(Stocks, Crypto, ETFs)"]
  C --> D["Computational Process<br>Evaluate 13 LLM backbone models<br>across varying market conditions"]
  D --> E["Key Outcomes<br>Standardized benchmark framework<br>Established accessible evaluation platform<br>Assessed reasoning capabilities"]

Cited References (103)

  1. Attention is all you need (2017)
  2. Gpt-4 technical report (2023) — link
  3. Reformer: The efficient transformer (2020) — link
  4. The handbook of news analytics in finance (2011)
  5. Noisy news in business cycles (2017)
  6. Stochastic optimal control: the discrete-time case (1996)
  7. Phi-3 technical report: A highly capable language model locally on your phone (2024) — link
  8. The official Meta Llama 3 GitHub site (2023) — link
and 95 more
  1. The Llama 3 Herd of Models (2024) — link
  2. Yi: Open foundation models by 01. ai (2024) — link
  3. Palmyra-Fin-70B-32k: a powerful LLM designed for Finance (2024)
  4. DeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback for Reinforcement Learning and Monte-Carlo Tree Search (2024) — link
  5. Qwen2.5 (2024) — link
  6. Introducing the next generation of Claude (2024) — link
  7. Introducing ChatGPT (2022) — link
  8. Gemma (2024) — link
  9. Gemini: a family of highly capable multimodal models (2023) — link
  10. Mixtral of Experts (2024) — link
  11. Language models are few-shot learners (2020)
  12. How can we know what language models know? (2020)
  13. Llama 2: Open foundation and fine-tuned chat models (2023) — link
  14. Training language models to follow instructions with human feedback (2022)
  15. Emergent abilities of large language models (2022) — link
  16. Finetuned language models are zero-shot learners (2021) — link
  17. Multitask prompted training enables zero-shot task generalization (2021) — link
  18. Reasoning with language model is planning with world model (2023) — link
  19. Art: Automatic multi-step reasoning and tool-use for large language models (2023) — link
  20. Sparks of artificial general intelligence: Early experiments with gpt-4 (2023) — link
  21. Planning with large language models for code generation (2023) — link
  22. Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change (2024)
  23. Leveraging pre-trained large language models to construct and utilize world models for model-based task planning (2023)
  24. On the planning abilities of large language models-a critical investigation (2023)
  25. Large language models still can’t plan (a benchmark for llms on planning and reasoning about change) (2022) — link
  26. LLMs Can’t Plan, But Can Help Planning in LLM-Modulo Frameworks (2024) — link
  27. Chain-of-thought prompting elicits reasoning in large language models (2022)
  28. Generative agents: Interactive simulacra of human behavior (2023)
  29. Voyager: An open-ended embodied agent with large language models (2023) — link
  30. The rise and potential of large language model based agents: A survey (2023) — link
  31. A survey on large language model based autonomous agents (2024)
  32. Language models as agent models (2022) — link
  33. Cognitive architectures for language agents (2023) — link
  34. Convfinqa: Exploring the chain of numerical reasoning in conversational finance question answering (2022) — link
  35. Finbert: A pre-trained financial language representation model for financial text mining (2021)
  36. Finbert: A pretrained language model for financial communications (2020) — link
  37. Finbert: Financial sentiment analysis with pre-trained language models (2019) — link
  38. FinBERT: A large language model for extracting information from financial text (2023)
  39. FinBERT–MRC: financial named entity recognition using BERT under the machine reading comprehension paradigm (2023)
  40. When flue meets flang: Benchmarks and large pre-trained language model for financial domain (2022) — link
  41. Fingpt: Democratizing internet-scale data for financial large language models (2023) — link
  42. Pixiu: A large language model, instruction data and evaluation benchmark for finance (2023) — link
  43. Bloomberggpt: A large language model for finance (2023) — link
  44. Investlm: A large language model for investment using financial domain instruction tuning — → 2309.13064
  45. Llms to the moon? reddit market sentiment analysis with large language models (2023)
  46. Learning to generate explainable stock predictions using self-reflective large language models — → 2402.03659
  47. Can large language models beat wall street? unveiling the potential of ai in stock selection — → 2401.03737
  48. FinMem: A performance-enhanced LLM trading agent with layered memory and character design (2024)
  49. FinAgent: A Multimodal Foundation Agent for Financial Trading: Tool-Augmented, Diversified, and Generalist — → 2402.18485
  50. FinRobot: An Open-Source AI Agent Platform for Financial Applications using Large Language Models — → 2405.14767
  51. Api-bank: A comprehensive benchmark for tool-augmented llms (2023) — link
  52. T-eval: Evaluating the tool utilization capability step by step (2023) — link
  53. NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classes (2023) — link
  54. BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (2024) — link
  55. Agent-Pro: Learning to Evolve via Policy-Level Reflection and Optimization (2024) — link
  56. WebArena: A Realistic Web Environment for Building Autonomous Agents (2023) — link
  57. Evaluating Language-Model Agents on Realistic Autonomous Tasks (2023) — link
  58. Agentbench: Evaluating llms as agents (2023) — link
  59. The finben: An holistic financial benchmark for large language models (2024) — link
  60. When AI Meets Finance (StockAgent): Large Language Model-based Stock Trading in Simulated Real-world Environments (2024)
  61. TradeMaster: a holistic quantitative trading platform empowered by reinforcement learning (2024)
  62. Risk Management and Financial Institutions (2007)
  63. The Sharpe Ratio (1994)
  64. Volatility Tests and Efficient Markets: A Review Essay (1988)
  65. Downside Risk (2003)
  66. Portfolio selection (2008)
  67. Quantitative equity investing: Techniques and strategies (2010)
  68. Adaptive quantitative trading: An imitative deep reinforcement learning approach (2020)
  69. Deep reinforcement learning approach for trading automation in the stock market (2022)
  70. Bayes in the age of intelligent machines (2023) — link
  71. Crypto News Dataset (2023) — link
  72. NIFTY Financial News Headlines Dataset — → 2405.09747
  73. LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Models (2024) — link
  74. Large language model based multi-agents: A survey of progress and challenges (2024) — link
  75. Determinants of llm-assisted decision-making (2024) — link
  76. Accurate stock movement prediction with self-supervised learning from sparse noisy tweets (2022)
  77. Deep attentive learning for stock movement prediction from social media text and company correlations (2020)
  78. Multi-source data driven cryptocurrency price movement prediction and portfolio optimization (2023)
  79. CNN-based multivariate data analysis for bitcoin trend prediction (2021)
  80. Trade the Event: Corporate Events Detection for News-Based Event-Driven Trading (2021) — link
  81. The effect of ETFs on financial markets: a literature review (2020)
  82. CryptoTrade: A Reflective LLM-based Agent to Guide Zero-shot Cryptocurrency Trading (2024)
  83. FinCon: A Synthesized LLM Multi-Agent System with Conceptual Verbal Reinforcement for Enhanced Financial Decision Making (2024) — link
  84. Cryptocurrency trading: a comprehensive survey (2022)
  85. Cryptocurrencies and stock market indices. Are they related? (2020)
  86. Project portfolio management in practice and in context (2013)
  87. Stock2vec: An embedding to improve predictive models for companies (2022) — link
  88. Demystifying the effect of the news (shocks) on crypto market volatility (2023)
  89. How Have ETFs Changed Market Macro Efficiency and Risk Structure? (2022)
  90. Exchange-traded funds and the new dynamics of investing (2016)
  91. Foundations of portfolio theory (1991)
  92. Fingpt: Open-source financial large language models — → 2306.06031
  93. Generative Agents: Interactive Simulacra of Human Behavior (2023) — link
  94. Replication and analysis of Ebbinghaus’ forgetting curve (2015)
  95. Portfolio selection (2011)

7 of 103 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.