Paper: arXiv 2403.15062

Abstract

With the recent development of large language models (LLMs), models that focus on certain domains and languages have been discussed for their necessity. There is also a growing need for benchmarks to evaluate the performance of current LLMs in each domain. Therefore, in this study, we constructed a benchmark comprising multiple tasks specific to the Japanese and financial domains and performed benchmark measurements on some models. Consequently, we confirmed that GPT-4 is currently outstanding, and that the constructed benchmarks function effectively. According to our analysis, our benchmark can differentiate benchmark scores among models in all performance ranges by combining tasks with different difficulties.

Complexity vs Empirical Score

  • Math Complexity: 1.5/10
  • Empirical Rigor: 6.0/10
  • Quadrant: Street Traders — practical and empirical, lighter on theory

Why this score: The paper focuses on benchmark construction and model evaluation using existing datasets and public resources, showing high empirical rigor through data handling and backtest-like evaluations, but the mathematics involved is relatively low-level (mostly classification metrics and data processing).

Research Flowchart

  flowchart TD
  A["Research Goal:\nConstruct Japanese Financial Benchmark for LLMs"] --> B["Methodology:\nDevelop Multi-Task Dataset (QA, Generation, Summarization)"]
  B --> C["Data & Inputs:\nJapanese Financial Documents & Financial QA"]
  C --> D["Computational Process:\nEvaluate GPT-4 & Existing Models\nusing constructed tasks"]
  D --> E["Outcome 1:\nGPT-4 Outperforms Other Models"]
  D --> F["Outcome 2:\nBenchmark Effectively Differentiates\nModel Performance Across Domains"]

Cited References (42)

  1. ChatGPT (2023)
  2. GPT-4 Technical Report (2023) — link
  3. A framework for few-shot language model evaluation (2021) — link
  4. Is it All Hype? ChatGPT’s Performance and Disruptive Potential in the Accounting and Auditing Industries (2023) — link
  5. Capabilities of GPT-4 on Medical Challenge Problems (2023)
  6. ChatGPT by OpenAI: The End of Litigation Lawyers? (2023) — link
  7. ChatGPT Goes to Law School (2023) — link
  8. Attention Is All You Need (2017)
and 34 more
  1. PaLM 2 Technical Report (2023)
  2. Llama 2: Open Foundation and Fine-Tuned Chat Models (2023)
  3. Improving Language Understanding by Generative Pre-Training (2018)
  4. Language Models are Unsupervised Multitask Learners (2019)
  5. Language Models are Few-Shot Learners (2020)
  6. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019) — link
  7. Bard (2023)
  8. Dolly (2023)
  9. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality (2023)
  10. LLaMA: Open and Efficient Foundation Language Models (2023)
  11. BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022)
  12. PaLM: Scaling Language Modeling with Pathways (2022)
  13. BloombergGPT: A Large Language Model for Finance (2023)
  14. Fin-LLAMA: Efficient Finetuning of Quantized LLMs for Finance (2023)
  15. FinGPT: Open-Source Financial Large Language Models (2023)
  16. Instruct-FinGPT: Financial Sentiment Analysis by Instruction Tuning of General-Purpose Large Language Models (2023)
  17. From Base to Conversational: Japanese Instruction Dataset and Tuning Large Language Models (2023)
  18. JMedLoRA: Medical Domain Adaptation on Japanese Large Language Models using Instruction-tuning (2023)
  19. llm-japanese-dataset v0: Construction of Japanese Chat Dataset for Large Language Models and its Methodology (2023) — link
  20. JP Language Model Evaluation Harness (2023)
  21. Gemini: a family of highly capable multimodal models (2023) — link
  22. Constructing and Analyzing Domain-Specific Language Model for Financial Text Mining (2023) — link
  23. llm-jp-eval (2024) — link
  24. Distributed Representations of Words and Phrases and their Compositionality (2013) — link
  25. Related Stocks Selection with Data Collaboration Using Text Mining (2019) — link
  26. Deep Contextualized Word Representations (2018) — link
  27. Bidirectional recurrent neural networks (1997) — link
  28. Universal Language Model Fine-tuning for Text Classification (2018) — link
  29. SciBERT: A Pretrained Language Model for Scientific Text (2019)
  30. Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction (2021)
  31. Construction and Validation of a Pre-Training and Additional Pre-Training Financial Language Model [in Japanese] (2022) — link
  32. Don’t stop pretraining: Adapt language models to domains and tasks (2020) — link
  33. chABSA: Aspect Based Sentiment Analysis dataset in Japanese (2018)
  34. Can ChatGPT pass the JCPA exam?: Challenge for the short-answer method test on Auditing (2023)

0 of 42 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.