Paper: arXiv 2502.11433

Abstract

Large language models (LLMs) fine-tuned on multimodal financial data have demonstrated impressive reasoning capabilities in various financial tasks. However, they often struggle with multi-step, goal-oriented scenarios in interactive financial markets, such as trading, where complex agentic approaches are required to improve decision-making. To address this, we propose \textsc{“FLAG-Trader”}, a unified architecture integrating linguistic processing (via LLMs) with gradient-driven reinforcement learning (RL) policy optimization, in which a partially fine-tuned LLM acts as the policy network, leveraging pre-trained knowledge while adapting to the financial domain through parameter-efficient fine-tuning. Through policy gradient optimization driven by trading rewards, our framework not only enhances LLM performance in trading but also improves results on other financial-domain tasks. We present extensive empirical evidence to validate these enhancements.

Complexity vs Empirical Score

  • Math Complexity: 7.5/10
  • Empirical Rigor: 8.0/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper employs advanced mathematical concepts such as Markov Decision Processes, policy gradients, and parameter-efficient fine-tuning, but also presents extensive empirical evidence, including backtested trading performance metrics and comparisons with baseline strategies.

Research Flowchart

  flowchart TD
  A["Research Goal: Improve LLM Decision-Making for Multi-Step Financial Trading"] --> B["Data Input: Multimodal Financial Data"]
  B --> C["Methodology: FLAG-Trader Architecture"]
  C --> D["LLM as Policy Network<br/>Parameter-Efficient Fine-Tuning"]
  C --> E["Gradient-Based RL<br/>Policy Gradient Optimization"]
  D --> F["Computational Process:<br/>Trading Reward Signal"]
  E --> F
  F --> G["Key Findings/Outcomes:<br/>Enhanced Trading Performance<br/>Improved General Financial Reasoning"]

Cited References (79)

  1. Proximal policy optimization algorithms (2017) — link
  2. Probability inequalities (2010)
  3. Human-level control through deep reinforcement learning (2015)
  4. On-line Q-learning using connectionist systems (1994)
  5. Learning from delayed rewards (1989)
  6. Reinforcement learning for trading (1998)
  7. Simple statistical gradient-following algorithms for connectionist reinforcement learning (1992)
  8. Trust Region Policy Optimization (2015) — link
and 71 more
  1. Asynchronous Methods for Deep Reinforcement Learning (2016) — link
  2. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor (2018)
  3. Continuous control with deep reinforcement learning (2015) — link
  4. Deterministic policy gradient algorithms (2014)
  5. FinRL-Meta: Market environments and benchmarks for data-driven financial reinforcement learning (2022)
  6. FinRL: Deep reinforcement learning framework to automate trading in quantitative finance (2021)
  7. An algorithm for trading and portfolio management using q-learning and sharpe ratio maximization (2000)
  8. Adaptive stock trading with dynamic asset allocation using reinforcement learning (2006)
  9. A multi-agent Q-learning framework for optimizing stock trading systems (2002)
  10. Reinforcement learning for high-frequency market making (2018)
  11. Learning FX trading strategies with FQI and persistent actions (2021)
  12. Practical deep reinforcement learning approach for stock trading (2018) — link
  13. Deep reinforcement learning for financial trading using price trailing (2019)
  14. A tabular sarsa-based stock market agent (2020)
  15. Improving financial trading decisions using deep Q-learning: Predicting the number of shares, action strategies, and transfer learning (2019)
  16. MetaTrader: An reinforcement learning approach integrating diverse policies for portfolio optimization (2022)
  17. DeepScalper: A risk-aware reinforcement learning framework to capture fleeting intraday trading opportunities (2022)
  18. Using data augmentation based reinforcement learning for daily stock trading (2020)
  19. Fingpt: Open-source financial large language models — → 2306.06031
  20. Decision transformer: Reinforcement learning via sequence modeling (2021)
  21. Trading with the momentum transformer: An intelligent and interpretable architecture (2021) — link
  22. The finben: An holistic financial benchmark for large language models (2024) — link
  23. Open-finllms: Open multimodal large language models for financial applications — → 2408.11878
  24. Time-llm: Time series forecasting by reprogramming large language models (2023) — link
  25. Cryptotrade: A reflective llm-based agent to guide zero-shot cryptocurrency trading (2024)
  26. TradingGPT: Multi-agent system with layered memory and distinct characters for enhanced financial trading performance — → 2309.03736
  27. Fincon: A synthesized llm multi-agent system with conceptual verbal reinforcement for enhanced financial decision making (2025)
  28. What do llms know about financial markets? a case study on reddit market sentiment analysis (2023)
  29. FinMem: A performance-enhanced LLM trading agent with layered memory and character design (2024)
  30. Text mining for market prediction: A systematic review (2014)
  31. Deep Reinforcement Learning for Trading (2019) — link
  32. An automated FX trading system using adaptive reinforcement learning (2006)
  33. Deep direct reinforcement learning for financial signal representation and trading (2016)
  34. A multi-scale temporal feature aggregation convolutional neural network for portfolio management (2019)
  35. A multi-objective deep reinforcement learning approach for stock index future’s intraday trading (2017)
  36. Relation-aware transformer for portfolio policy learning (2021)
  37. Commission fee is not enough: A hierarchical reinforced framework for portfolio management (2021)
  38. Cost-sensitive portfolio selection via deep reinforcement learning (2020)
  39. DeepTrader: a deep reinforcement learning approach for risk-return balanced portfolio management with market conditions Embedding (2021)
  40. Reinforcement-learning based portfolio management with augmented asset movement prediction states (2020)
  41. Adversarial deep reinforcement learning in portfolio management (2018) — link
  42. An end-to-end optimal trade execution framework based on proximal policy optimization (2021)
  43. Deep reinforcement learning for automated stock trading: An ensemble strategy (2020)
  44. Quantitative day trading from natural language using reinforcement learning (2021)
  45. A novel deep reinforcement learning framework with BiLSTM-Attention networks for algorithmic trading (2024)
  46. Deep reinforcement learning for financial trading using multi-modal features (2024)
  47. Recent advances in reinforcement learning in finance (2023)
  48. Reinforcement learning for quantitative trading (2023)
  49. INVESTORBENCH: A Benchmark for Financial Decision-Making Tasks with LLM-based Agent — → 2412.18174
  50. Risk Management and Financial Institutions (2007)
  51. Large language models as generalizable policies for embodied tasks (2023)
  52. Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning (2024) — link
  53. Convfinqa: Exploring the chain of numerical reasoning in conversational finance question answering (2022) — link
  54. Finbert: A pre-trained financial language representation model for financial text mining (2021)
  55. Finbert: A pretrained language model for financial communications (2020) — link
  56. Finbert: Financial sentiment analysis with pre-trained language models (2019) — link
  57. FinBERT: A large language model for extracting information from financial text (2023)
  58. FinBERT–MRC: financial named entity recognition using BERT under the machine reading comprehension paradigm (2023)
  59. When flue meets flang: Benchmarks and large pre-trained language model for financial domain (2022) — link
  60. Fingpt: Democratizing internet-scale data for financial large language models (2023) — link
  61. Pixiu: A large language model, instruction data and evaluation benchmark for finance (2023) — link
  62. Bloomberggpt: A large language model for finance (2023) — link
  63. Investlm: A large language model for investment using financial domain instruction tuning — → 2309.13064
  64. Learning to generate explainable stock predictions using self-reflective large language models — → 2402.03659
  65. Can large language models beat wall street? unveiling the potential of ai in stock selection — → 2401.03737
  66. FinAgent: A Multimodal Foundation Agent for Financial Trading: Tool-Augmented, Diversified, and Generalist — → 2402.18485
  67. FinRobot: An Open-Source AI Agent Platform for Financial Applications using Large Language Models — → 2405.14767
  68. When AI Meets Finance (StockAgent): Large Language Model-based Stock Trading in Simulated Real-world Environments (2024)
  69. TradeMaster: a holistic quantitative trading platform empowered by reinforcement learning (2024)
  70. Fincon: A synthesized llm multi-agent system with conceptual verbal reinforcement for enhanced financial decision making (2024) — link
  71. An overview of machine learning, deep learning, and reinforcement learning-based techniques in quantitative finance: recent progress and challenges (2023)

9 of 79 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.