Paper: arXiv 2404.10555

Abstract

Large language models (LLMs) are now widely used in various fields, including finance. However, Japanese financial-specific LLMs have not been proposed yet. Hence, this study aims to construct a Japanese financial-specific LLM through continual pre-training. Before tuning, we constructed Japanese financial-focused datasets for continual pre-training. As a base model, we employed a Japanese LLM that achieved state-of-the-art performance on Japanese financial benchmarks among the 10-billion-class parameter models. After continual pre-training using the datasets and the base model, the tuned model performed better than the original model on the Japanese financial benchmarks. Moreover, the outputs comparison results reveal that the tuned model’s outputs tend to be better than the original model’s outputs in terms of the quality and length of the answers. These findings indicate that domain-specific continual pre-training is also effective for LLMs. The tuned model is publicly available on Hugging Face.

Complexity vs Empirical Score

  • Math Complexity: 1.5/10
  • Empirical Rigor: 8.5/10
  • Quadrant: Street Traders — practical and empirical, lighter on theory

Why this score: The paper uses advanced ML concepts like transformer architectures and continual pre-training, but the focus is on implementation details (data prep, training hyperparameters) and empirical results (benchmarks, public model release).

Research Flowchart

  flowchart TD
  A["Research Goal: Construct<br>Japanese Financial LLM"] --> B["Dataset Construction"]
  B --> C["Select Base Model<br>Japanese LLM 10B"]
  C --> D["Continual Pre-training<br>on Financial Data"]
  D --> E{"Evaluation"}
  E --> F["Key Findings: Outcomes"]
  
  subgraph F["Key Findings: Outcomes"]
      F1["Benchmark Improvement<br>Better than original model"]
      F2["Output Quality<br>Higher quality & length"]
      F3["Public Release<br>Hugging Face"]
  end
  
  B -->|Input: Japanese<br>Financial Texts| D
  C -->|Input: 10B Parameter<br>Japanese LLM| D

Cited References (50)

  1. ChatGPT (2023)
  2. GPT-4 Technical Report (2023) — link
  3. A framework for few-shot language model evaluation (2021) — link
  4. Is it All Hype? ChatGPT’s Performance and Disruptive Potential in the Accounting and Auditing Industries (2023) — link
  5. Capabilities of GPT-4 on Medical Challenge Problems (2023)
  6. ChatGPT by OpenAI: The End of Litigation Lawyers? (2023) — link
  7. ChatGPT Goes to Law School (2023) — link
  8. Attention Is All You Need (2017)
and 42 more
  1. PaLM 2 Technical Report (2023)
  2. Llama 2: Open Foundation and Fine-Tuned Chat Models (2023)
  3. Improving Language Understanding by Generative Pre-Training (2018)
  4. Language Models are Unsupervised Multitask Learners (2019)
  5. Language Models are Few-Shot Learners (2020)
  6. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019) — link
  7. Bard (2023)
  8. Dolly (2023)
  9. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality (2023)
  10. LLaMA: Open and Efficient Foundation Language Models (2023)
  11. BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022)
  12. PaLM: Scaling Language Modeling with Pathways (2022)
  13. BloombergGPT: A Large Language Model for Finance (2023)
  14. Fin-LLAMA: Efficient Finetuning of Quantized LLMs for Finance (2023)
  15. FinGPT: Open-Source Financial Large Language Models (2023)
  16. Instruct-FinGPT: Financial Sentiment Analysis by Instruction Tuning of General-Purpose Large Language Models (2023)
  17. From Base to Conversational: Japanese Instruction Dataset and Tuning Large Language Models (2023)
  18. JMedLoRA: Medical Domain Adaptation on Japanese Large Language Models using Instruction-tuning (2023)
  19. llm-japanese-dataset v0: Construction of Japanese Chat Dataset for Large Language Models and its Methodology (2023) — link
  20. JP Language Model Evaluation Harness (2023)
  21. Gemini: a family of highly capable multimodal models (2023) — link
  22. Constructing and Analyzing Domain-Specific Language Model for Financial Text Mining (2023) — link
  23. llm-jp-eval (2024) — link
  24. Distributed Representations of Words and Phrases and their Compositionality (2013)
  25. Related Stocks Selection with Data Collaboration Using Text Mining (2019) — link
  26. Deep Contextualized Word Representations (2018) — link
  27. Bidirectional recurrent neural networks (1997) — link
  28. Universal Language Model Fine-tuning for Text Classification (2018) — link
  29. SciBERT: A Pretrained Language Model for Scientific Text (2019)
  30. Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction (2021)
  31. Construction and Validation of a Pre-Training and Additional Pre-Training Financial Language Model [in Japanese] (2022) — link
  32. Don’t stop pretraining: Adapt language models to domains and tasks (2020) — link
  33. Construction of a Japanese Financial Benchmark for Large Language Models (2024) — link
  34. Lora: Low-rank adaptation of large language models (2021) — link
  35. Finetuned language models are zero-shot learners (2021) — link
  36. Deep reinforcement learning from human preferences (2017)
  37. Direct preference optimization: Your language model is secretly a reward model (2024)
  38. Lima: Less is more for alignment (2024)
  39. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters (2020)
  40. Accelerate: Training and inference at scale made simple, efficient and adaptable. (2022)
  41. Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling (2024) — link
  42. Can ChatGPT pass the JCPA exam?: Challenge for the short-answer method test on Auditing (2023)

0 of 50 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.