Your AI, Not Your View: The Bias of LLMs in Investment Analysis

Paper: arXiv 2507.20957

Authors: Hoyoung Lee, Junhyuk Seo, Suhwan Park, Junhyeong Lee, Wonbin Ahn, Chanyeol Choi, Alejandro Lopez-Lira, Yongjae Lee

Abstract

In finance, Large Language Models (LLMs) face frequent knowledge conflicts arising from discrepancies between their pre-trained parametric knowledge and real-time market data. These conflicts are especially problematic in real-world investment services, where a model’s inherent biases can misalign with institutional objectives, leading to unreliable recommendations. Despite this risk, the intrinsic investment biases of LLMs remain underexplored. We propose an experimental framework to investigate emergent behaviors in such conflict scenarios, offering a quantitative analysis of bias in LLM-based investment analysis. Using hypothetical scenarios with balanced and imbalanced arguments, we extract the latent biases of models and measure their persistence. Our analysis, centered on sector, size, and momentum, reveals distinct, model-specific biases. Across most models, a tendency to prefer technology stocks, large-cap stocks, and contrarian strategies is observed. These foundational biases often escalate into confirmation bias, causing models to cling to initial judgments even when faced with increasing counter-evidence. A public leaderboard benchmarking bias across a broader set of models is available at https://linqalpha.com/leaderboard

Complexity vs Empirical Score

• Math Complexity: 4.0/10
• Empirical Rigor: 6.0/10
• Quadrant: Street Traders — practical and empirical, lighter on theory

Why this score: The paper focuses on experimental methodology and bias quantification using LLM prompting, with limited advanced mathematics, but demonstrates high empirical rigor through a structured framework, hypothetical scenario testing, and a public leaderboard benchmark.

Research Flowchart

  flowchart TD
  A["Research Goal: Quantify LLM Investment Biases in Knowledge Conflicts"] --> B["Methodology: Hypothetical Scenario Testing"]
  B --> C{"Data: Balanced vs. Imbalanced<br/>Sector/Size/Momentum Arguments"}
  C --> D["Process: Extract & Measure<br/>Latent Bias & Persistence"]
  D --> E["Computation: Statistical Analysis<br/>of Model Responses"]
  E --> F["Key Outcomes: Distinct Model-Specific Biases"]
  F --> G["Findings: Tech, Large-Cap,<br/>Contrarian Preference + Confirmation Bias"]

Cited References (36)

1. Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts (2023)
2. What evidence do language models find convincing? (2024) — link
3. Cognitive bias in decision-making with LLMs (2024) — link
4. LLM Agents Can Be Choice-Supportive Biased Evaluators: An Empirical Study (2025)
5. Fincon: A synthesized llm multi-agent system with conceptual verbal reinforcement for enhanced financial decision making (2024)
6. What Is Seen Cannot Be Unseen: The Disruptive Effect of Knowledge Conflict on Large Language Models (2025) — link
7. Blinded by generated contexts: How language models merge generated and retrieved contexts when knowledge conflicts? (2024) — link
8. Exposing product bias in llm investment recommendation (2025) — link

8 of 36 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.