Open-FinLLMs: Open Multimodal Large Language Models for Financial Applications

Paper: arXiv 2408.11878

Abstract

Financial LLMs hold promise for advancing financial tasks and domain-specific applications. However, they are limited by scarce corpora, weak multimodal capabilities, and narrow evaluations, making them less suited for real-world application. To address this, we introduce \textit{“Open-FinLLMs”}, the first open-source multimodal financial LLMs designed to handle diverse tasks across text, tabular, time-series, and chart data, excelling in zero-shot, few-shot, and fine-tuning settings. The suite includes FinLLaMA, pre-trained on a comprehensive 52-billion-token corpus; FinLLaMA-Instruct, fine-tuned with 573K financial instructions; and FinLLaVA, enhanced with 1.43M multimodal tuning pairs for strong cross-modal reasoning. We comprehensively evaluate Open-FinLLMs across 14 financial tasks, 30 datasets, and 4 multimodal tasks in zero-shot, few-shot, and supervised fine-tuning settings, introducing two new multimodal evaluation datasets. Our results show that Open-FinLLMs outperforms afvanced financial and general LLMs such as GPT-4, across financial NLP, decision-making, and multi-modal tasks, highlighting their potential to tackle real-world challenges. To foster innovation and collaboration across academia and industry, we release all codes (https://anonymous.4open.science/r/PIXIU2-0D70/B1D7/LICENSE) and models under OSI-approved licenses.

Complexity vs Empirical Score

• Math Complexity: 4.0/10
• Empirical Rigor: 8.5/10
• Quadrant: Street Traders — practical and empirical, lighter on theory

Why this score: The paper focuses on engineering a suite of open-source LLMs, quantifying performance across extensive tasks and datasets with a lack of advanced mathematical derivations. Its strength lies in heavy implementation, large-scale data curation, and comprehensive benchmarking against real-world models like GPT-4.

Research Flowchart

  flowchart TD
  Goal["Research Goal<br/>Develop open-source, multimodal LLMs for financial tasks"]
  Data["Data & Corpora<br/>• 52B-token text corpus<br/>• 573K financial instructions<br/>• 1.43M multimodal image-text pairs"]
  Pretrain["Methodology: Pre-training<br/>FinLLaMA model on corpus"]
  Instruct["Methodology: Instruction Tuning<br/>FinLLaMA-Instruct on instructions"]
  Multi["Methodology: Multimodal Tuning<br/>FinLLaVA on image-text pairs"]
  Eval["Evaluation<br/>14 tasks, 30 datasets, 4 multimodal tasks<br/>(Zero/Few/Supervised Shot)"]
  Outcome["Key Findings<br/>Outperforms GPT-4 & financial LLMs<br/>Releases: Models & Code (OSI)"]

  Goal --> Data
  Data --> Pretrain
  Data --> Instruct
  Data --> Multi
  Pretrain --> Instruct
  Instruct --> Multi
  Multi --> Eval
  Eval --> Outcome

Cited References (206)

1. Gpt-4 technical report (2023) — link
2. Fino1: On the Transferability of Reasoning Enhanced LLMs to Finance (2025) — link
3. Doremi: Optimizing data mixtures speeds up language model pretraining (2024)
4. MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning (2023) — link
5. Data-Juicer: A One-Stop Data Processing System for Large Language Models (2024)
6. FinBen: A Holistic Financial Benchmark for Large Language Models (2024) — link
7. The FineWeb datasets: Decanting the Web for the finest text data at scale (2024) — link
8. Llama 2: Open foundation and fine-tuned Chat models (2023) — link

4 of 206 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.