Paper: arXiv 2402.11728

Abstract

In this paper, we investigate the influence of claims in analyst reports and earnings calls on financial market returns, considering them as significant quarterly events for publicly traded companies. To facilitate a comprehensive analysis, we construct a new financial dataset for the claim detection task in the financial domain. We benchmark various language models on this dataset and propose a novel weak-supervision model that incorporates the knowledge of subject matter experts (SMEs) in the aggregation function, outperforming existing approaches. We also demonstrate the practical utility of our proposed model by constructing a novel measure of optimism. Here, we observe the dependence of earnings surprise and return on our optimism measure. Our dataset, models, and code are publicly (under CC BY 4.0 license) available on GitHub.

Complexity vs Empirical Score

  • Math Complexity: 2.5/10
  • Empirical Rigor: 8.0/10
  • Quadrant: Street Traders — practical and empirical, lighter on theory

Why this score: The paper is primarily focused on NLP and dataset construction with benchmarking language models and public data release, showing high empirical rigor, while the mathematics involved (statistics for weak-supervision aggregation) is relatively low in complexity.

Research Flowchart

  flowchart TD
  A["Research Goal: Determine influence of claims<br>on market returns & create claim detection model"] --> B["Create New Financial Dataset"]
  B --> C["Develop Novel Weak-Supervision Model<br>incorporating SME knowledge"]
  C --> D["Benchmark against other<br>Language Models"]
  D --> E["Construct Novel<br>Optimism Measure"]
  E --> F["Analyze Earnings Surprise<br>vs. Optimism"]
  F --> G["Key Findings: Claims impact markets,<br>Proposed model outperforms<br>Optimism correlates with earnings"]

Cited References (63)

  1. Llama 2: Open Foundation and Fine-Tuned Chat Models (2023) — link
  2. Trillion Dollar Words: A New Financial Dataset, Task & Market Analysis — → 2305.07972
  3. When FLUE Meets FLANG: Benchmarks and Large Pretrained Language Model for Financial Domain (2022) — link
  4. Analysts’ forecast optimism: The effects of managers’ incentives on analysts’ forecasts (2022) — link
  5. WRENCH: A Comprehensive Benchmark for Weak Supervision (2021) — link
  6. FinQA: A Dataset of Numerical Reasoning over Financial Data (2021) — link
  7. Persuading Investors: A Video-Based Study (2021) — link
  8. Multimodal Machine Learning for Credit Modeling (2021) — link
and 55 more
  1. skweak: Weak Supervision Made Easy for NLP (2021) — link
  2. Green Algorithms: Quantifying the Carbon Footprint of Computation (2020) — link
  3. VolTAGE: Volatility Forecasting via Text-Audio Fusion with Graph Convolution Networks for Earnings Calls (2020) — link
  4. NumClaim: Investor’s Fine-grained Claim Detection (2020) — link
  5. MAEC: A Multimodal Aligned Earnings Conference Call Dataset for Financial Risk Prediction (2020) — link
  6. Fine-Tuning Pre-trained Language Model with Weak Supervision: A Contrastive-Regularized Self-Training Approach (2020) — link
  7. How to Talk When a Machine is Listening: Corporate Disclosure in the Age of AI (2020) — link
  8. FinBERT: A Pre-trained Financial Language Representation Model for Financial Text Mining (2020) — link
  9. Measuring Corporate Culture Using Machine Learning (2020) — link
  10. FinBERT: A Pretrained Language Model for Financial Communications (2020) — link
  11. Man Versus Machine: A Comparison of Robo-Analyst and Traditional Research Analyst Investment Recommendations (2020) — link
  12. PyTorch: An Imperative Style, High-Performance Deep Learning Library (2019) — link
  13. FinBERT: Financial Sentiment Analysis with Pre-trained Language Models (2019) — link
  14. RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019) — link
  15. Numeral Attachment with Auxiliary Tasks (2019) — link
  16. Snuba: Automating Weak Supervision to Label Training Data (2018) — link
  17. Weakly-Supervised Neural Text Classification (2018) — link
  18. WWW'18 Open Challenge: Financial Opinion Mining and Question Answering (2018) — link
  19. Snorkel: Rapid Training Data Creation with Weak Supervision (2017) — link
  20. A Multilayer Perceptron based Ensemble Technique for Fine-grained Financial Sentiment Analysis (2017) — link
  21. Attention is All you Need (2017) — link
  22. Deep learning for financial sentiment analysis on finance news providers (2016) — link
  23. GloVe: Global Vectors for Word Representation (2014) — link
  24. Good debt or bad debt: Detecting semantic orientations in economic texts (2013) — link
  25. A Survey of Crowdsourcing Systems (2011) — link
  26. A Survey on Transfer Learning (2010) — link
  27. Do Analysts Herd? An Analysis of Recommendations and Market Reactions (2010) — link
  28. Making Sense of Cents: An Examination of Firms That Marginally Miss or Beat Analyst Forecasts (2009) — link
  29. Distant supervision for relation extraction without labeled data (2009) — link
  30. Conflict of interest and the credibility of underwriter analyst recommendations (1999) — link
  31. Long Short-Term Memory (1997) — link
  32. Bidirectional recurrent neural networks (1997) — link
  33. ANALYSTS DECISIONS AS PRODUCTS OF A MULTITASK ENVIRONMENT (1993) — link
  34. Retail Investors and Analysts (2024) — link
  35. . Falcon-40B: an open large language model with state-of-the-art performance (2023)
  36. Do Managers Walk the Talk on Environmental and Social Issues? (2021) — link
  37. ReGAL: Rule-Generative Active Learning for Model-in-the-Loop Weak Supervision (2020)
  38. Transform-ers: State-of-the-art natural language processing (2020)
  39. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019) — link
  40. Investment Banking Relationships and Analyst Affiliation Bias: The Impact of Global Settlement on Sanctioned and Non-Sanctioned Banks (2014)
  41. Semi-Supervised Learning (2009) — link
  42. Burr Settles (2009)
  43. Comparing the stock recommendation performance of investment banks and independent research firms (2006) — link
    1. Measuring firm-level inflation exposure: A deep learning approach
  45. Phrases often provided definitive information about a given sentence in a document and in most cases they had a fairly consistent linguistic composition
  46. 2023b. Finer: Financial named entity recognition dataset and weak-supervision model
  47. a bid to capture the effect of a few other verb forms indicative of a probabilistic event, we
  48. by us while choosing the rules to weak-supervision mode. All rules were post detailed analysis of sample
  49. no contamination issue in the weak-supervision model and it is generalizable, we get
    1. Chat-gpt in finance: Applications, challenges, and solutions
    1. Actune: Uncertainty-based active self-training for active fine-tuning of pretrained language models
    1. Closed ai models make bad baselines
  53. The alternate adoption of phrase matching was to identify in-claim sentences
  54. robustness of our model on the dataset annotated by annotators who don’t have knowledge of the rules used in the weak-supervision model
    1. Falcon-40B: an open large language model

1 of 63 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.