EDINET-Bench: Evaluating LLMs on Complex Financial Tasks using Japanese Financial Statements

Paper: arXiv 2506.08762

Authors: Issa Sugiura, Takashi Ishida, Taro Makino, Chieko Tazuke, Takanori Nakagawa, Kosuke Nakago, David Ha

Abstract

Financial analysis presents complex challenges that could leverage large language model (LLM) capabilities. However, the scarcity of challenging financial datasets, particularly for Japanese financial data, impedes academic innovation in financial analytics. As LLMs advance, this lack of accessible research resources increasingly hinders their development and evaluation in this specialized domain. To address this gap, we introduce EDINET-Bench, an open-source Japanese financial benchmark designed to evaluate the performance of LLMs on challenging financial tasks including accounting fraud detection, earnings forecasting, and industry prediction. EDINET-Bench is constructed by downloading annual reports from the past 10 years from Japan’s Electronic Disclosure for Investors’ NETwork (EDINET) and automatically assigning labels corresponding to each evaluation task. Our experiments reveal that even state-of-the-art LLMs struggle, performing only slightly better than logistic regression in binary classification for fraud detection and earnings forecasting. These results highlight significant challenges in applying LLMs to real-world financial applications and underscore the need for domain-specific adaptation. Our dataset, benchmark construction code, and evaluation code is publicly available to facilitate future research in finance with LLMs.

Complexity vs Empirical Score

• Math Complexity: 2.0/10
• Empirical Rigor: 8.0/10
• Quadrant: Street Traders — practical and empirical, lighter on theory

Why this score: The paper focuses on dataset creation, benchmarking, and comparing LLM performance with logistic regression using real-world financial data, with minimal mathematical derivations.

Research Flowchart

  flowchart TD
  A["Research Goal: Evaluate LLMs on complex Japanese financial tasks"] --> B["Methodology: Construct EDINET-Bench"]
  B --> C["Data Source: 10 Years of EDINET Annual Reports"]
  C --> D["Tasks: Fraud Detection, Earnings Forecasting, Industry Prediction"]
  D --> E["Evaluation: SOTA LLMs vs. Baselines"]
  E --> F["Outcome: LLMs struggle vs. simple baselines"]
  F --> G["Conclusion: Need for domain-specific adaptation"]

Cited References (88)

1. Automated design of agentic systems (2025)
2. Predicting industry sectors from financial statements: An illustration of machine learning in accounting research (2022)
3. Financial statement analysis and the prediction of stock returns (1989)
4. Predicting future earnings changes using machine learning and detailed financial data (2022)
5. GDP (current US$) (2025) — link
6. LiveBench: A Challenging, Contamination-Limited LLM Benchmark (2025)
7. An Evolved Universal Transformer Memory (2024) — link
8. Construction of domain-specified japanese large language model for finance through continual pre-training — → 2404.10555

3 of 88 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.