Paper: arXiv 2410.14059

Abstract

This paper introduces the UCFE: User-Centric Financial Expertise benchmark, an innovative framework designed to evaluate the ability of large language models (LLMs) to handle complex real-world financial tasks. UCFE benchmark adopts a hybrid approach that combines human expert evaluations with dynamic, task-specific interactions to simulate the complexities of evolving financial scenarios. Firstly, we conducted a user study involving 804 participants, collecting their feedback on financial tasks. Secondly, based on this feedback, we created our dataset that encompasses a wide range of user intents and interactions. This dataset serves as the foundation for benchmarking 11 LLMs services using the LLM-as-Judge methodology. Our results show a significant alignment between benchmark scores and human preferences, with a Pearson correlation coefficient of 0.78, confirming the effectiveness of the UCFE dataset and our evaluation approach. UCFE benchmark not only reveals the potential of LLMs in the financial domain but also provides a robust framework for assessing their performance and user satisfaction.

Complexity vs Empirical Score

  • Math Complexity: 2.0/10
  • Empirical Rigor: 8.0/10
  • Quadrant: Street Traders — practical and empirical, lighter on theory

Why this score: The paper introduces a new evaluation benchmark with a user study, dataset creation, and a rigorous correlation analysis (Pearson r=0.78), but contains almost no advanced mathematics or formal modeling.

Research Flowchart

  flowchart TD
  A["Research Goal<br>Evaluate LLM Financial Expertise<br>User-Centric Approach"] --> B["Methodology: User Study<br>804 Participants & Feedback"]
  B --> C["Methodology: Dataset Creation<br>Hybrid User-Intent & Interaction Data"]
  C --> D["Benchmarking Process<br>11 LLM Services Evaluated"]
  D --> E["Computational Process<br>LLM-as-Judge Methodology"]
  E --> F["Key Outcome<br>High Alignment: Pearson r=0.78"]
  F --> G["Final Contribution<br>UCFE Framework & Benchmark"]

Cited References (70)

  1. A User-Centric Benchmark for Evaluating Large Language Models (2024) — link
  2. FinEval: A Chinese Financial Domain Knowledge Evaluation Benchmark for Large Language Models (2023) — link
  3. InternLM2 Technical Report (2024) — link
  4. CFGPT: Chinese financial assistant with large language model (2023) — link
  5. GPT-4 Technical Report (2024) — link
  6. Using ELO ratings for match result prediction in association football (2010)
  7. The LDBC Financial Benchmark (2023) — link
  8. G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment (2023) — link
and 62 more
  1. SuperCLUE-Fin: Graded Fine-Grained Analysis of Chinese LLMs on Diverse Financial Tasks and Applications (2024) — link
  2. BBT-Fin: Comprehensive Construction of Chinese Financial Domain Pre-trained Language Model, Corpus and Benchmark (2023) — link
  3. FinanceBench: A New Benchmark for Financial Question Answering (2023) — link
  4. CFGPT: Chinese Financial Assistant with Large Language Model (2023) — link
  5. CFBenchmark: Chinese Financial Assistant Benchmark for Large Language Model (2024) — link
  6. Palmyra-Fin-70B-32k: a powerful LLM designed for Finance (2024)
  7. Qwen2.5: A Party of Foundation Models (2024) — link
  8. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model (2024) — link
  9. XuanYuan 2.0: A Large Chinese Financial Chat Model with Hundreds of Billions Parameters (2023) — link
  10. Llama 3 Model Card (2024) — link
  11. Open-FinLLMs: Open Multimodal Large Language Models for Financial Applications (2024) — link
  12. FinGPT: Open-Source Financial Large Language Models (2023) — link
  13. A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges — → 2406.11903
  14. No Language is an Island: Unifying Chinese and English in Financial Large Language Models, Instruction Data, and Benchmarks (2024) — link
  15. Split and Merge: Aligning Position Biases in Large Language Model based Evaluators (2023) — link
  16. Can Large Language Models be Used to Provide Psychological Counselling? An Analysis of GPT-4-Generated Responses Using Role-play Dialogues (2024) — link
  17. Bloomberggpt: A large language model for finance (2023) — link
  18. FinBERT: A large language model for extracting information from financial text (2023)
  19. Financial statement analysis with large language models — → 2407.17866
  20. Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge (2024) — link
  21. Bias and Toxicity in Role-Play Reasoning (2024) — link
  22. Humans or llms as the judge? a study on judgement biases (2024) — link
  23. Machine learning: an applied econometric approach (2017)
  24. Financial markets and growth: An overview (1993)
  25. More than just noise? Examining the information content of stock microblogs on financial markets (2018)
  26. Mental accounting and consumer choice (2008)
  27. The framing of decisions and the psychology of choice (1981)
  28. A size-of-loss model for the negatively skewed insurance claims data: applications, risk analysis using different methods and statistical forecasting (2024)
  29. Can chatgpt forecast stock price movements? return predictability and large language models — → 2304.07619
  30. Comparative Analysis of NLP-Based Models for Company Classification (2024)
  31. Leveraging vision-language models for granular market change prediction — → 2301.10166
  32. A survey on large language model (llm) security and privacy: The good, the bad, and the ugly (2024)
  33. What’s in Your" Safe" Data?: Identifying Benign Data that Breaks Safety (2024) — link
  34. Challenging the appearance of machine intelligence: Cognitive bias in LLMs and Best Practices for Adoption (2023) — link
  35. Can LLMs Learn Macroeconomic Narratives from Social Media? (2024) — link
  36. Generative AI based virtual assistant for reconciliation research (2024)
  37. Ploutos: Towards interpretable stock movement prediction with financial large language model — → 2403.00782
  38. Generative AI for Business Strategy: Using Foundation Models to Create Business Strategy Tools (2023) — link
  39. Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templates (2024) — link
  40. When flue meets flang: Benchmarks and large pre-trained language model for financial domain (2022) — link
  41. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi (2024)
  42. MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark (2024) — link
  43. Scaling Laws for Neural Language Models (2020) — link
  44. Observational Scaling Laws and the Predictability of Language Model Performance (2024) — link
  45. Over-Reasoning and Redundant Calculation of Large Language Models (2024) — link
  46. Pixiu: A large language model, instruction data and evaluation benchmark for finance (2023) — link
  47. Large language models and generative AI in finance: an analysis of ChatGPT, Bard, and Bing AI (2023)
  48. Learning to generate explainable stock predictions using self-reflective large language models — → 2402.03659
  49. Are ChatGPT and GPT-4 general-purpose solvers for financial text analytics? A study on several typical tasks (2023) — link
  50. R-judge: Benchmarking safety risk awareness for llm agents (2024) — link
  51. EUCA: The end-user-centered explainable AI framework (2021) — link
  52. Exploring the nature and implementation process of user-centric business models (2011)
  53. Enterprise design thinking: An investigation on user-centered design processes in large corporations (2021)
  54. Explainable Robo-Advisors: Empirical Investigations to Specify and Evaluate a User-Centric Taxonomy of Explanations in the Financial Domain. (2022)
  55. A human-centric framework for robotic task learning and optimization (2023)
  56. Artificial intelligence-enhanced predictive insights for advancing financial inclusion: A human-centric ai-thinking approach (2020)
  57. Responsible and human centric AI-based insurance advisors (2023)
  58. User-Centric Explainability in Fintech Applications (2021)
  59. AI-driven UX/UI design: Empirical research and applications in FinTech (2024)
  60. Chatbot arena: An open platform for evaluating llms by human preference (2024) — link
  61. TwinMarket: A Scalable Behavioral and SocialSimulation for Financial Markets (2025) — link
  62. Econagent: large language model-empowered agents for simulating macroeconomic activities (2024)

6 of 70 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.