Paper: arXiv 2412.10906

Abstract

The rapid growth of the financial sector and the rising focus on Environmental, Social, and Governance (ESG) considerations highlight the need for advanced NLP tools. However, open-source LLMs proficient in both finance and ESG domains remain scarce. To address this gap, we introduce SusGen-30K, a category-balanced dataset comprising seven financial NLP tasks and ESG report generation, and propose TCFD-Bench, a benchmark for evaluating sustainability report generation. Leveraging this dataset, we developed SusGen-GPT, a suite of models achieving state-of-the-art performance across six adapted and two off-the-shelf tasks, trailing GPT-4 by only 2% despite using 7-8B parameters compared to GPT-4’s 1,700B. Based on this, we propose the SusGen system, integrated with Retrieval-Augmented Generation (RAG), to assist in sustainability report generation. This work demonstrates the efficiency of our approach, advancing research in finance and ESG.

Complexity vs Empirical Score

  • Math Complexity: 3.0/10
  • Empirical Rigor: 8.5/10
  • Quadrant: Street Traders — practical and empirical, lighter on theory

Research Flowchart

  flowchart TD
  A["Research Goal: Develop open-source LLM for finance & ESG"] --> B["Methodology: Dataset Creation & Model Training"]
  B --> C["Data/Inputs: SusGen-30K dataset"]
  C --> D["Computational Process: SusGen-GPT 7-8B training"]
  D --> E["Findings: State-of-the-art performance"]
  E --> F["Outcome: SusGen System w/ RAG for report generation"]

Cited References (274)

  1. The Theory of Parsing, Translation and Compiling (1972)
  2. Publications Manual (1983)
  3. Alternation (1981) — link
  4. Scalable training of L1 -regularized log-linear models (2007)
  5. Algorithms on Strings, Trees and Sequences (1997)
  6. Yara Parser: A Fast and Accurate Dependency Parser (2015) — link
  7. GPT-4 Technical Report (2023) — link
  8. Sparks of artificial general intelligence: Early experiments with gpt-4 (2023) — link
and 266 more
  1. Larger language models do in-context learning differently (2023) — link
  2. The Wall Street Neophyte: A Zero-Shot Analysis of ChatGPT Over MultiModal Stock Movement Prediction Challenges — → 2304.05351
  3. Can chatgpt forecast stock price movements? return predictability and large language models — → 2304.07619
  4. Are ChatGPT and GPT-4 General-Purpose Solvers for Financial Text Analytics? An Examination on Several Typical Tasks (2023) — link
  5. Holistic evaluation of language models (2022) — link
  6. Measuring massive multitask language understanding (2020) — link
  7. Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models (2023)
  8. BBT-Fin: Comprehensive Construction of Chinese Financial Domain Pre-trained Language Model, Corpus and Benchmark (2023) — link
  9. Transformers: State-of-the-Art Natural Language Processing (2020) — link
  10. Memory Sandbox: Transparent and Interactive Memory Management for Conversational Agents (2023) — link
  11. CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes (2023) — link
  12. Social Simulacra: Creating Populated Prototypes for Social Computing Systems (2022) — link
  13. Cognitive Architectures for Language Agents (2023) — link
  14. A Survey on Large Language Model based Autonomous Agents (2023) — link
  15. Improving Language Understanding by Generative Pre-Training (2018) — link
  16. RecAgent: A Novel Simulation Paradigm for Recommender Systems (2023) — link
  17. FinGPT: Instruction Tuning Benchmark for Open-Source Large Language Models in Financial Datasets (2023) — link
  18. An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labels (2022) — link
  19. FinMem: A Performance-Enhanced LLM Trading Agent with Layered Memory and Character Design (2023) — link
  20. The Cattell-Horn-Carroll model of intelligence. (2012)
  21. CHC theory and the human cognitive abilities project: Standing on the shoulders of the giants of psychometric intelligence research (2009)
  22. Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Texts (2013) — link
  23. Instruction Tuning for Large Language Models: A Survey (2023) — link
  24. InvestLM: A Large Language Model for Investment using Financial Domain Instruction Tuning (2023) — link
  25. XuanYuan 2.0: A Large Chinese Financial Chat Model with Hundreds of Billions Parameters (2023) — link
  26. F in BART : A Pre-trained Seq2seq Language Model for C hinese Financial Tasks (2023) — link
  27. CGCE: A Chinese Generative Chat Evaluation Benchmark for General and Financial Domains (2023) — link
  28. CFBenchmark: Chinese Financial Assistant Benchmark for Large Language Model (2023) — link
  29. DISC-FinLLM: A Chinese Financial Large Language Model based on Multiple Experts Fine-tuning (2023) — link
  30. Large Language Models in Finance: A Survey (2023) — link
  31. FinBERT: Financial Sentiment Analysis with Pre-trained Language Models (2019) — link
  32. FinBERT: A Pretrained Language Model for Financial Communications (2020) — link
  33. FinBERT: A Pre-trained Financial Language Representation Model for Financial Text Mining (2020) — link
  34. Future climate change significantly alters interannual wheat yield variability over half of harvested areas (2021) — link
  35. ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators (2020) — link
  36. A Survey of Large Language Models in Finance (FinLLMs) (2024) — link
  37. WWW'18 Open Challenge: Financial Opinion Mining and Question Answering (2018) — link
  38. S em E val-2017 Task 5: Fine-Grained Sentiment Analysis on Financial Microblogs and News (2017) — link
  39. Impact of News on the Commodity Market: Dataset and Results (2020) — link
  40. Domain Adaption of Named Entity Recognition to Support Credit Risk Assessment (2015) — link
  41. F i NER : Financial Numeric Entity Recognition for XBRL Tagging (2022) — link
  42. FinRED: A Dataset for Relation Extraction in Financial Domain (2023) — link
  43. The Financial Document Causality Detection Shared Task ( F in C ausal 2020) (2020) — link
  44. FinQA: A Dataset of Numerical Reasoning over Financial Data (2022) — link
  45. ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering (2022) — link
  46. MAEC: A Multimodal Aligned Earnings Conference Call Dataset for Financial Risk Prediction (2020) — link
  47. Multilingual and Cross-Lingual Intent Detection from Spoken Data (2021) — link
  48. M ulti F in: A Dataset for Multilingual Financial NLP (2023) — link
  49. Global Table Extractor (GTE): A Framework for Joint Table Identification and Cell Structure Recognition Using Visual Context (2021)
  50. Trade the Event: Corporate Events Detection for News-Based Event-Driven Trading (2021) — link
  51. StockEmotions: Discover Investor Emotions for Financial Sentiment Analysis and Multivariate Time Series (2023) — link
  52. CFGPT: Chinese Financial Assistant with Large Language Model (2023) — link
  53. MONOPOLY: Financial Prediction from MONetary POLicY Conference Videos Using Multimodal Cues (2022) — link
  54. Gemini: a family of highly capable multimodal models (2023) — link
  55. GLM: General Language Model Pretraining with Autoregressive Blank Infilling (2022)
  56. Qwen Technical Report (2023) — link
  57. Baichuan 2: Open Large-scale Language Models (2023) — link
  58. Internlm: A multilingual language model with progressively enhanced capabilities (2023)
  59. The falcon series of open language models (2023) — link
  60. Mixtral of experts (2024) — link
  61. Code llama: Open foundation models for code (2023) — link
  62. Select and Trade: Towards Unified Pair Trading with Hierarchical Reinforcement Learning — → 2301.10724
  63. Finetuned language models are zero-shot learners (2021) — link
  64. Bloom: A 176b-parameter open-access multilingual language model (2022) — link
  65. Multitask Prompted Training Enables Zero-Shot Task Generalization (2022)
  66. The flan collection: Designing data and methods for effective instruction tuning (2023) — link
  67. Mengzi: Towards lightweight yet ingenious pre-trained models for chinese (2021) — link
  68. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019)
  69. Good debt or bad debt: Detecting semantic orientations in economic texts (2014)
  70. Impact of news on the commodity market: Dataset and results (2021)
  71. Domain adaption of named entity recognition to support credit risk assessment (2015)
  72. Www'18 open challenge: financial opinion mining and question answering (2018)
  73. Self-Instruct: Aligning Language Model with Self Generated Instructions (2022) — link
  74. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality (2023) — link
  75. FinQA: A Dataset of Numerical Reasoning over Financial Data (2021)
  76. Decoupled weight decay regularization (2017) — link
  77. GPT-NeoX-20B : An Open-Source Autoregressive Language Model (2022) — link
  78. Opt: Open pre-trained transformer language models (2022) — link
  79. Solving quantitative reasoning problems with language models (2022) — link
  80. Training verifiers to solve math word problems (2021) — link
  81. Accurate Stock Movement Prediction with Self-supervised Learning from Sparse Noisy Tweets (2022)
  82. Stock movement prediction from tweets and historical prices (2018)
  83. Hybrid deep sequential modeling for social text-driven stock prediction (2018)
  84. BELLE: Be Everyone’s Large Language model Engine (2023)
  85. Exploring the Impact of Instruction Data Scaling on Large Language Models: An Empirical Study on Real-World Use Cases (2023) — link
  86. Select and Trade: Towards Unified Pair Trading with Hierarchical Reinforcement Learning (2023)
  87. Enhancing stock movement prediction with adversarial training (2018) — link
  88. Accurate multivariate stock movement prediction via data-axis transformer with multi-level contexts (2021)
  89. Semeval-2017 task 5: Fine-grained sentiment analysis on financial microblogs and news (2017)
  90. FinanceBench: A New Benchmark for Financial Question Answering (2023) — link
  91. Bizbench: A quantitative reasoning benchmark for business and finance (2023) — link
  92. CALM-Bench: A Multi-task Benchmark for Evaluating Causality-Aware Language Models (2023)
  93. Disc-finllm: A chinese financial large language model based on multiple experts fine-tuning (2023) — link
  94. Fin-Eva Version 1.0 (2023)
  95. Trillion Dollar Words: A New Financial Dataset, Task & Market Analysis (2023) — link
  96. Fine-Grained Argument Understanding with BERT Ensemble Techniques: A Deep Dive into Financial Sentiment Analysis (2023)
  97. MultiFin: A Dataset for Multilingual Financial NLP (2023)
  98. Generating plausible counterfactual explanations for deep transformers in financial text classification (2020) — link
  99. Multi-Lingual ESG Issue Identification (2023)
  100. Finer: Financial named entity recognition dataset and weak-supervision model (2023) — link
  101. FinRED: A dataset for relation extraction in financial domain (2022)
  102. Financial document causality detection shared task (fincausal 2020) (2020) — link
  103. A probabilistic interpretation of precision, recall and F-score, with implication for evaluation (2005)
  104. Complementarity, F -score, and NLP Evaluation (2016) — link
  105. TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance (2021) — link
  106. Financial Numeric Extreme Labelling: A dataset and benchmarking (2023)
  107. Textual analogy parsing: What’s shared and what’s compared among analogous facts (2018) — link
  108. How are We Detecting Inconsistent Method Names? An Empirical Study from Code Review Perspective (2023) — link
  109. Ectsum: A new benchmark dataset for bullet point summarization of long earnings call transcripts (2022) — link
  110. Rouge: A package for automatic evaluation of summaries (2004)
  111. Bertscore: Evaluating text generation with bert (2019) — link
  112. Bartscore: Evaluating generated text as text generation (2021)
  113. Statlog (German Credit Data) (1994)
  114. Statlog (Australian Credit Approval)
  115. Empowering many, biasing a few: Generalist credit scoring through large language models (2023) — link
  116. The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation (2020)
  117. Strategic management decision-making in a complex world: quantifying, understanding, and using trade-offs (2017)
  118. A monthly effect in stock returns (1987)
  119. The sharpe ratio (1998)
  120. Forecasting the equity premium: Do deep neural network models work? (2023)
  121. Tracking the fin trade: genetic stock identification in western Atlantic scalloped hammerhead sharks Sphyrna lewini (2009)
  122. Maximum drawdown (2004)
  123. Fingpt: Democratizing internet-scale data for financial large language models (2023) — link
  124. Dynamic Datasets and Market Environments for Financial Reinforcement Learning (2023) — link
  125. FinRL-Meta: Market Environments and Benchmarks for Data-Driven Financial Reinforcement Learning (2022) — link
  126. D'olares or Dollars? Unraveling the Bilingual Prowess of Financial LLMs Between Spanish and English (2024) — link
  127. LAiW: A Chinese Legal Large Language Models Benchmark (2024) — link
  128. Back to the Future: Towards Explainable Temporal Reasoning with Large Language Models (2023) — link
  129. Empowering Many, Biasing a Few: Generalist Credit Scoring through Large Language Models (2024) — link
  130. Mastering Pair Trading with Risk-Aware Recurrent Reinforcement Learning (2023) — link
  131. Qlora: Efficient finetuning of quantized llms (2024)
  132. Decoupled Weight Decay Regularization (2019) — link
  133. PaLM 2 Technical Report (2023) — link
  134. The Choice of Textual Knowledge Base in Automated Claim Checking (2023) — link
  135. chatClimate: Grounding Conversational AI in Climate Science (2023) — link
  136. Aggregate confusion: The divergence of ESG ratings (2022)
  137. Towards Continual Knowledge Learning of Language Models (2022)
  138. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? (2021) — link
  139. It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners (2021) — link
  140. Large Language Models are Zero-Shot Reasoners (2023) — link
  141. Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm (2021) — link
  142. A Survey on Automated Fact-Checking (2022) — link
  143. The Choice of Textual Knowledge Base in Automated Claim Checking (2023) — link
  144. Fact Checking: Task definition and dataset construction (2014) — link
  145. Attention is All you Need (2017) — link
  146. Ethical and social risks of harm from Language Models (2021) — link
  147. Improving language models by retrieving from trillions of tokens (2022) — link
  148. Towards a framework for evaluation and design of conversational agents (2016)
  149. The impact of chatbot characteristics on user satisfaction and conversational performance (2020)
  150. Climate-fever: A dataset for verification of real-world climate claims (2020) — link
  151. A Framework for Understanding and Evaluating Automated Systems (2020)
  152. In bot we trust: A new methodology of chatbot performance measures (2019) — link
  153. ChatGPT: A Large-scale Generative Language Model for Conversational AI (2022)
  154. SAINT $\mathplus$: Integrating Temporal Features for EdNet Correctness Prediction (2021) — link
  155. Towards Healthy AI: Large Language Models Need Therapists Too (2023) — link
  156. A Bibliometric Review of Large Language Models Research from 2017 to 2023 (2023) — link
  157. Towards a Human-like Open-Domain Chatbot (2020) — link
  158. ClimateBert: A Pretrained Language Model for Climate-Related Text (2022) — link
  159. Machine-learning-based evidence and attribution mapping of 100,000 climate impact studies (2021) — link
  160. Analyzing Sustainability Reports Using Natural Language Processing (2020) — link
  161. Low-Resource Adaptation of Open-Domain Generative Chatbots (2022) — link
  162. DERA: Enhancing Large Language Model Completions with Dialog-Enabled Resolving Agents (2023) — link
  163. HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in HuggingFace (2023) — link
  164. BloombergGPT: A Large Language Model for Finance (2023) — link
  165. CFBenchmark: Chinese Financial Assistant Benchmark for Large Language Model (2024) — link
  166. Enhancing Large Language Models with Climate Resources (2023) — link
  167. The importance of accurate and up-to-date information in the context of climate change (2020)
  168. Climate change and cities: challenges ahead (2021)
  169. Climate Change 2022: Mitigation of Climate Change (2022)
  170. Climate Change 2022: Impacts, Adaptation, and Vulnerability (2022)
  171. Climate Change 2021: The Physical Science Basis (2021)
  172. Survey of Hallucination in Natural Language Generation (2023) — link
  173. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019) — link
  174. Prompting Large Language Models with Answer Heuristics for Knowledge-based Visual Question Answering (2023) — link
  175. Leveraging Large Language Models for Multiple Choice Question Answering (2023) — link
  176. Self-Prompting Large Language Models for Open-Domain QA (2022) — link
  177. Plug-and-Play Adaptation for Continuously-updated QA (2022) — link
  178. Exploiting cloze questions for few shot text classification and natural language inference (2020) — link
  179. Visual chatGPT : Talking, drawing and editing with visual foundation models (2023) — link
  180. How good are GPT models at machine translation? A comprehensive evaluation (2023) — link
  181. Progressive-hint prompting improves reasoning in large language models (2023) — link
  182. Chain of thought prompting elicits reasoning in large language models (2022) — link
  183. Making pre-trained language models better few-shot learners (2020) — link
  184. Toolformer: Language models can teach themselves to use tools (2023) — link
  185. PIXIU: A Large Language Model, Instruction Data and Evaluation Benchmark for Finance (2023) — link
  186. Mistral 7B (2023) — link
  187. The Llama 3 Herd of Models (2024) — link
  188. Dictionary-based phrase-level prompting of large language models for machine translation (2023) — link
  189. Large language models are human-level prompt engineers (2022) — link
  190. Multitask prompted training enables zero-shot task generalization (2021) — link
  191. Cheap talk in corporate climate commitments: The role of active institutional ownership, signaling, materiality, and sentiment (2022)
  192. ClimateBERT: A Pretrained Language Model for Climate-Related Text (2022)
  193. Environmental Claim Detection (2023)
  194. Cheap talk and cherry-picking: What ClimateBert has to say on corporate climate risk disclosures (2022)
  195. Towards Climate Awareness in NLP Research (2022) — link
  196. Language models are few-shot learners (2020)
  197. Language Models are Unsupervised Multitask Learners (2019)
  198. Sustainable development through climate action (2019) — link
  199. Quantifying the human cost of global warming (2023)
  200. Toward a new understanding of environmental and financial performance through corporate social responsibility, green innovation, and sustainable development (2023)
  201. A sustainable development pathway for climate action within the UN 2030 Agenda (2021) — link
  202. Training language models to follow instructions with human feedback (2022)
  203. The Promise and Peril of Generative AI (2023)
  204. Attention is all you need (2017)
  205. Llama: Open and efficient foundation language models (2023) — link
  206. Quantifying the economic risks of climate change (2017)
  207. GPT-4 Technical Report (2023) — link
  208. Recommendations of the Task Force on Climate-Related Financial Disclosures (2017)
  209. Implementing the Recommendations of the TCFD (2021)
  210. Climate Change and Financial Stability (2021)
  211. The role of CO2 capture and utilization in mitigating climate change (2017)
  212. Task force on climate-related financial disclosures: Overview (2017)
  213. Machine-learning-based evidence and attribution mapping of 100,000 climate impact studies (2021)
  214. InstructGPT: AI for Generating Instructions (2023)
  215. RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019) — link
  216. Dangers of AI: LLMs, Inaccuracy, and the Need for Responsibility (2021) — link
  217. Global Warming of 1.5°C. An IPCC Special Report on the impacts of global warming of 1.5°C above pre-industrial levels and related global greenhouse gas emission pathways, in the context of strengthening the global response to the threat of climate change (2018)
  218. Global Warming: The Complete Briefing (2015)
  219. The Economics of Climate Change: The Stern Review (2007)
  220. Uncertainty estimates in regional and global observed temperature changes: A new data set from 1850 (2006)
  221. Training a single AI model can emit as much carbon as five cars in their lifetimes (2019)
  222. Lora: Low-rank adaptation of large language models (2021) — link
  223. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (2022) — link
  224. 8-bit Optimizers via Block-wise Quantization (2022)
  225. Pause Giant AI Experiments: An Open Letter (2023)
  226. The real costs of financial efficiency when some information is soft (2016)
  227. Palm: Scaling language modeling with pathways (2022) — link
  228. TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIs (2023) — link
  229. Ask BERT : How regulatory disclosure of transition and physical climate risks affects the CDS term structure (2020)
  230. Vicuna: An open-source chatbot impressing GPT -4 with 90%* chatgpt quality (2023)
  231. Instruction tuning with GPT -4 (2023) — link
  232. Large Language Models are not Fair Evaluators (2023) — link
  233. Lima: Less is more for alignment (2023) — link
  234. Automatic Evaluation of Attribution by Large Language Models (2023) — link
  235. Evaluating Verifiability in Generative Search Engines (2023) — link
  236. Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback (2023) — link
  237. Self- Checker : Plug -and- Play Modules for Fact - Checking with Large Language Models (2023) — link
  238. Judging LLM -as-a-judge with MT - Bench and Chatbot Arena (2023) — link
  239. Active Learning with Feedback on Both Features and Instances (2006)
  240. Active Learning for Graph Neural Networks via Node Feature Propagation (2021) — link
  241. Interscript: A dataset for interactive learning of scripts through error feedback (2021) — link
  242. Learning to repair: Repairing model output errors after deployment using a dynamic memory of feedback (2022) — link
  243. Large Language Models Are State -of-the- Art Evaluators of Translation Quality (2023) — link
  244. How is ChatGPT’s behavior changing over time? (2023) — link
  245. Llama 2: Open Foundation and Fine-Tuned Chat Models (2023) — link
  246. Stanford Alpaca: An Instruction-following LLaMA model (2023)
  247. WHEN FLUE MEETS FLANG: Benchmarks and Large Pre-trained Language Model for Financial Domain (2022) — link
  248. FinGPT: Open-Source Financial Large Language Models (2023)
  249. Instruct-FinGPT: Financial Sentiment Analysis by Instruction Tuning of General-Purpose Large Language Models (2023)
  250. Enhancing Financial Sentiment Analysis via Retrieval Augmented Large Language Models — → 2310.04027
  251. FinGPT: Instruction Tuning Benchmark for Open-Source Large Language Models in Financial Datasets (2023)
  252. Data-centric FinGPT: Democratizing Internet-scale Data for Financial Large Language Models (2023)
  253. The FinBen: An Holistic Financial Benchmark for Large Language Models (2024) — link
  254. Common 7B Language Models Already Possess Strong Math Capabilities (2024) — link
  255. RAFT: Adapting Language Model to Domain Specific RAG (2024) — link
  256. Scaling Laws for Neural Language Models (2020) — link
  257. Training Verifiers to Solve Math Word Problems (2021) — link
  258. Self-training Large Language Models through Knowledge Detection (2024) — link
  259. Lora learns less and forgets less (2024) — link
  260. Evaluating Topic Model on Asymmetric and Multi-Domain Financial Corpus (2024) — link
  261. Large Language Models as Financial Data Annotators: A Study on Effectiveness and Efficiency (2024) — link
  262. CHATREPORT : Democratizing Sustainability Disclosure Analysis through LLM -based Tools (2023) — link
  263. Glitter or gold? Deriving structured insights from sustainability reports via large language models (2024)
  264. ESGReveal: An LLM-based approach for extracting structured data from ESG reports (2023) — link
  265. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments (2005)
  266. Bleu: a method for automatic evaluation of machine translation (2002)

4 of 274 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.