Paper: arXiv 2311.15180

Abstract

The impact of non-deterministic outputs from Large Language Models (LLMs) is not well examined for financial text understanding tasks. Through a compelling case study on investing in the US equity market via news sentiment analysis, we uncover substantial variability in sentence-level sentiment classification results, underscoring the innate volatility of LLM outputs. These uncertainties cascade downstream, leading to more significant variations in portfolio construction and return. While tweaking the temperature parameter in the language model decoder presents a potential remedy, it comes at the expense of stifled creativity. Similarly, while ensembling multiple outputs mitigates the effect of volatile outputs, it demands a notable computational investment. This work furnishes practitioners with invaluable insights for adeptly navigating uncertainty in the integration of LLMs into financial decision-making, particularly in scenarios dictated by non-deterministic information.

Complexity vs Empirical Score

  • Math Complexity: 2.5/10
  • Empirical Rigor: 6.0/10 (capped from 7.0: transaction costs not modelled; see penalty flags)
  • Quadrant: Street Traders — practical and empirical, lighter on theory

Why this score: The paper is empirically heavy with a case study on US equity market sentiment analysis using real LLMs and backtestable portfolio construction, but mathematically sparse, focusing on practical volatility measurement via outputs rather than advanced theory.

Research Flowchart

  flowchart TD
  A["Research Goal:<br>Quantify LLM Volatility Impact<br>on Financial Decision-Making"] --> B["Methodology:<br>News Sentiment Analysis Case Study"]
  B --> C["Input:<br>US Equity Market News Data"]
  C --> D["Process:<br>LLM Inference with<br>Variable Temperature Parameters"]
  D --> E["Outcome 1:<br>High Variability in<br>Sentence-Level Sentiment"]
  E --> F["Downstream Effect:<br>Volatile Portfolio Construction"]
  F --> G["Key Finding:<br>Uncertainty Cascades to Returns"]
  G --> H["Mitigation Strategies:<br>Low Temperature vs.<br>Computational Ensembling"]

Cited References (115)

  1. The patent holder’s dilemma: Buy, sell, or troll? (2007) — link
  2. Deciding equivalances among conjunctive aggregate queries (2007) — link
  3. Special issue: Digital Libraries (1996)
  4. Understanding Policy-Based Networking (2001)
  5. First-Order Dynamic Logic (1979) — link
  6. The title of book one (2007) — link
  7. The title of book two (2008) — link
  8. Achieving application requirements (1990) — link
and 107 more
  1. Statecarts in use: structured analysis and object-orientation (1998) — link
  2. The Art of Computer Programming, Vol. 1: Fundamental Algorithms (3rd. ed.) (1997)
  3. The Art of Computer Programming (1998)
  4. An experiment in bibliographic mark-up: Parsing metadata for XML export (2010) — link
  5. Catch me, if you can: Evading network signatures with web-based polymorphic worms (2007)
  6. Catch me, if you can: Evading network signatures with web-based polymorphic worms (2008)
  7. Catch me, if you can: Evading network signatures with web-based polymorphic worms (2009)
  8. Predicate Path expressions (1979) — link
  9. LOGICS of Programs: AXIOMATICS and DESCRIPTIVE POWER (1978)
  10. Optimal Motion Control of a Ground Vehicle (2003)
  11. Algorithms for Closest-Point Problems (Computational Geometry) (1985)
  12. Introduction to Bayesian Statistics (2001) — link
  13. CLIFFORD: a Maple 11 Package for Clifford Algebra Computations, version 11 (2007) — link
  14. Stats and Analysis (2006) — link
  15. A more perfect union (2008) — link
  16. The fountain of youth (2009)
  17. Solder man (2003) — link
  18. Interview with Bill Kinder: January 13, 2005 (2005) — link
  19. The Enabling of Digital Libraries (2008)
  20. (old) Finding minimum congestion spanning trees (2000) — link
  21. (new) Finding minimum congestion spanning trees (2000) — link
  22. (old) Distributed data source verification in wireless sensor networks (2009) — link
  23. (new) Distributed data source verification in wireless sensor networks (2009) — link
  24. Portalis: using competitive online interactions to support aid initiatives for the homeless (2008) — link
  25. Visual Basic 6: Design, Specification, and Objects with Other (1999)
  26. The Latex Web Companion: Integrating TEX, HTML, and XML (1999)
  27. Vertex Types in Book-Embeddings (1987)
  28. Vertex Types in Book-Embeddings (1987)
  29. CHI ‘08: CHI ‘08 extended abstracts on Human factors in computing systems (2008)
  30. Algorithms for Closest-Point Problems (Computational Geometry) (1985)
  31. IEEE TCSC Executive Committee (2004) — link
  32. Distributed systems (2nd Ed.) (1993)
  33. New Algorithms for Dependency-Directed Backtracking (Master’s thesis) (1986)
  34. New Algorithms for Dependency-Directed Backtracking (Master’s thesis) (1986)
  35. Seminumerical Algorithms (1981)
  36. E-commerce and cultural values (2001) — link
  37. Chapter 9 (2002) — link
  38. The implementation of electronic commerce in SMEs in Singapore (Incoll) (2003) — link
  39. E-commerce and cultural values - (InBook-num-in-chap) (2004) — link
  40. E-commerce and cultural values (Inbook-text-in-chap) (2005) — link
  41. E-commerce and cultural values (Inbook-num chap) (2006) — link
  42. A library-based synthesis methodology for reversible logic (2010)
  43. Synthesis of Reversible Circuit Using Cycle-Based Approach (2010)
  44. Algorithmic Enumeration of Ideal Classes for Quaternion Orders (2010) — link
  45. Chapter II: Notes on data structuring (1972) — link
  46. Transcript of question and answer session (1981) — link
  47. Go to statement considered harmful (1979) — link
  48. Three-dimensional virtual acoustic displays (1992) — link
  49. Managerial expert systems and organizational change: some critical research issues (1987) — link
  50. Simplified Structured COBOL with Microsoft/MicroFocus COBOL (1990)
  51. A multi-radio unification protocol for IEEE 802.11 wireless networks (2004)
  52. Wireless Sensor Networks: A Survey (2002)
  53. A Survey on Wireless Multimedia Sensor Networks (2007)
  54. SSCH : Slotted Seeded Channel Hopping for Capacity Improvement in IEEE 802.11 Ad-Hoc Wireless Networks (2004)
  55. XBOW Sensor Motes Specifications (2008)
  56. Overview of Sensor Networks (2004)
  57. CodeBlue : Sensor Networks for Medical Care (2008)
  58. Investigating Network Architectures for Body Sensor Networks (2007) — link
  59. Channel-Hopping Multiple Access (2000)
  60. Body Sensor Networks (2008)
  61. Mapping Powerlists onto Hypercubes (1994)
  62. Automatic Parallelization for Distributed-Memory Multiprocessing Systems (1989)
  63. User recovery and reversal in interactive systems (1984)
  64. Generalizing specifications for uniformly implemented loops (1985)
  65. Towards monolingual programming environments (1985)
  66. The \TeX book (1984)
  67. Distributed algorithms for finding centers and medians in networks (1984)
  68. \it \LaTeX : A Document Preparation System (1986)
  69. Program transformations in a denotational setting (1985)
  70. Seminumerical Algorithms (1981)
  71. A high-level approach to computer document formatting (1980)
  72. A multifrequency MAC specially designed for wireless sensor network applications (2010) — link
  73. Institutional members of the \TeX Users Group (2017) — link
  74. acmart— C lass for typesetting publications of ACM (2017) — link
  75. Reasoning About Naming Systems (1993) — link
  76. Babel, a Multilingual Style-Option System for Use with LaTeX’s Standard Document Styles (1991)
  77. A Methodology for Implementing Highly Concurrent Data Objects (1993) — link
  78. Publication quality tables in \LaTeX (2005)
  79. Using the amsthm Package (2015)
  80. R: A Language and Environment for Statistical Computing (2019) — link
  81. UMass Citation Field Extraction Dataset (2013) — link
  82. Citation concept analysis (CCA) —A new form of citation analysis revealing the usefulness of concepts for other researchers illustrated by two exemplary case studies including classic books by Thomas S.~Kuhn and Karl R.~Popper (2019) — link
  83. Learning Soft Linear Constraints with Application to Citation Field Extraction (2014) — link
  84. Maintaining Discrete Probability Distributions Optimally (1993)
  85. Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models (2023) — link
  86. Quantifying social biases using templates is unreliable (2022) — link
  87. Can ChatGPT predict future interest rate decisions? (2023)
  88. Can chatgpt forecast stock price movements? return predictability and large language models — → 2304.07619
  89. Deep probability estimation (2021) — link
  90. Dropout as a bayesian approximation: Representing model uncertainty in deep learning (2016)
  91. Simple and scalable predictive uncertainty estimation using deep ensembles (2017)
  92. What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variability (2023) — link
  93. Llama: Open and efficient foundation language models (2023) — link
  94. GPT-4 Technical Report (2023) — link
  95. Large language models can rate news outlet credibility (2023) — link
  96. Attention is all you need (2017)
  97. News Signals: An NLP Library for Text and Time Series (2023)
  98. Leancontext: Cost-efficient domain-specific question answering using llms (2023) — link
  99. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation (2023) — link
  100. Scilab — link
  101. Scilab — link
  102. The Computational Geometry Algorithms Library — link
  103. The Computational Geometry Algorithms Library — link
  104. The Parmap library — link
  105. The Parmap library — link
  106. BabyJ : from Object Based to Class Based Programming via Types (2003)
  107. tag-sound

1 of 115 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.