Paper: arXiv 2402.06698

Abstract

Financial market predictions utilize historical data to anticipate future stock prices and market trends. Traditionally, these predictions have focused on the statistical analysis of quantitative factors, such as stock prices, trading volumes, inflation rates, and changes in industrial production. Recent advancements in large language models motivate the integrated financial analysis of both sentiment data, particularly market news, and numerical factors. Nonetheless, this methodology frequently encounters constraints due to the paucity of extensive datasets that amalgamate both quantitative and qualitative sentiment analyses. To address this challenge, we introduce a large-scale financial dataset, namely, Financial News and Stock Price Integration Dataset (FNSPID). It comprises 29.7 million stock prices and 15.7 million time-aligned financial news records for 4,775 S&P500 companies, covering the period from 1999 to 2023, sourced from 4 stock market news websites. We demonstrate that FNSPID excels existing stock market datasets in scale and diversity while uniquely incorporating sentiment information. Through financial analysis experiments on FNSPID, we propose: (1) the dataset’s size and quality significantly boost market prediction accuracy; (2) adding sentiment scores modestly enhances performance on the transformer-based model; (3) a reproducible procedure that can update the dataset. Completed work, code, documentation, and examples are available at github.com/Zdong104/FNSPID. FNSPID offers unprecedented opportunities for the financial research community to advance predictive modeling and analysis.

Complexity vs Empirical Score

  • Math Complexity: 2.0/10
  • Empirical Rigor: 8.5/10
  • Quadrant: Street Traders — practical and empirical, lighter on theory

Why this score: The paper focuses primarily on dataset construction, empirical experiments, and reproducible code, with minimal advanced mathematical derivations beyond standard regression and model descriptions.

Research Flowchart

  flowchart TD
  A["Research Goal:<br>Integrate Sentiment & Quantitative Data<br>for Stock Prediction"] --> B["Data Acquisition:<br>Collect 15.7M News Records<br>& 29.7M Stock Prices"]
  B --> C["Data Construction:<br>Time-Align Sentiment & Price Data<br>(1999-2023, S&P 500)"]
  C --> D["Computational Process:<br>Train Transformer Models<br>with/without Sentiment Scores"]
  D --> E["Outcome 1:<br>Dataset Scale/Quality<br>Significantly Boosts Accuracy"]
  D --> F["Outcome 2:<br>Sentiment Scores<br>Modestly Enhance Performance"]
  D --> G["Outcome 3:<br>Reproducible Procedure<br>for Future Updates"]

Cited References (54)

  1. The cross-section of expected stock returns (1992)
  2. Some Empirical Tests of the Theory of Arbitrage Pricing (1983)
  3. Capital Asset Prices: A Theory of Market Equilibrium under Conditions of Risk (1964)
  4. Stock price prediction using reinforcement learning (2001) — link
  5. Adaptive stock trading strategies with deep reinforcement learning methods (2020) — link
  6. Stock price prediction: comparison of different moving average techniques using deep learning model (2024) — link
  7. Predicting stock market using machine learning: best and accurate way to know future stock prices (2023) — link
  8. A Comprehensive Comparative Study of Artificial Neural Network (ANN) and Support Vector Machines (SVM) on Stock Forecasting (2023) — link
and 46 more
  1. Deep Reinforcement Learning for Automated Stock Trading: An Ensemble Strategy (2021) — link
  2. Stock Market Prediction Using Machine Learning Classifiers and Social Media, News (2022) — link
  3. Can ChatGPT Forecast Stock Price Movements? Return Predictability and Large Language Models (2023) — link
  4. Stock Prediction Based on Deep Learning and its Application in Pairs Trading (2022) — link
  5. Design a neural network for time series financial forecasting: Accuracy and robustness analysis (2008)
  6. Predicting stock market using machine learning: best and accurate way to know future stock prices (2023)
  7. A stock price prediction method based on deep learning technology (2021)
  8. News sentiment and stock market volatility (2021)
  9. Stock Price Prediction using Sentiment Analysis and Deep Learning for Indian Markets (2022) — link
  10. ChatGPT (2023) — link
  11. Sentiment Analysis for Stock Price Prediction (2020) — link
  12. Stock Price Prediction Using News Sentiment Analysis (2019) — link
  13. One Fits All:Power General Time Series Analysis by Pretrained LM (2023) — link
  14. FinGPT: Instruction Tuning Benchmark for Open-Source Large Language Models in Financial Datasets (2023) — link
  15. Dynamic Datasets and Market Environments for Financial Reinforcement Learning (2024)
  16. The legacy of modern portfolio theory (2002)
  17. A network and machine learning approach to factor, asset, and blended allocation (2020)
  18. Dynamic connectedness between investors’ sentiment and asset prices: A comparison between major markets in Europe and USA (2023) — link
  19. Data Augmentation for Sentiment Analysis Using Sentence Compression-Based SeqGAN With Data Screening (2021) — link
  20. Effect of training set size on SVM and Naive Bayes for Twitter sentiment analysis (2015) — link
  21. GAN-BElectra: Enhanced Multi-class Sentiment Analysis with Limited Labeled Data (2022) — link
  22. The impact of different training data set on the accuracy of sentiment classification of Naïve Bayes technique (2017) — link
  23. Financial News Dataset from Bloomberg and Reuters (2015)
  24. Corpus of news articles of Lenta.Ru (2019)
  25. Finnhub Stock APIs
  26. Sentence-Level Sentiment Analysis of Financial News Using Distributed Text Representations and Multi-Instance Learning (2018) — link
  27. Leveraging Latent Economic Concepts and Sentiments in the News for Market Prediction (2021) — link
  28. A novel ensemble deep learning model for stock prediction based on stock prices and news (2020) — link
  29. Fine-Grained Sentiment Analysis on Financial Microblogs and News Headlines (2017) — link
  30. News Sentiment Impact Analysis (NSIA) Framework (2016) — link
  31. SEntFiN 1.0: Entity‐aware sentiment analysis for financial news (2022) — link
  32. Predicting Mergers and Acquisitions using Graph-based Deep Learning (2021)
  33. Exploring mutual information-based sentimental analysis with kernel-based extreme learning machine for stock prediction (2017) — link
  34. Layout guide for Journal of Physics: conference series using Microsoft Word (2023) — link
  35. Investigating Deep Stock Market Forecasting with Sentiment Analysis (2023) — link
  36. SemEval-2017 Task 5: Fine-Grained Sentiment Analysis on Financial Microblogs and News (2017) — link
  37. Text summarization and singular value decomposition (2005)
  38. Transforming sentiment analysis in the financial domain with ChatGPT (2023)
  39. ChatGPT: Jack of all trades, master of none (2023)
  40. Instruct-FinGPT: Financial Sentiment Analysis by Instruction Tuning of General-Purpose Large Language Models — → 2306.12659
  41. Algorithm and modeling of stock prices forecasting based on long short-term memory (LSTM) (2018)
  42. Financial time-series data analysis using deep convolutional neural networks (2016)
  43. Deep learning with gated recurrent unit networks for financial sequence predictions (2018)
  44. TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis (2023) — link
  45. Attention is all you need (2017)
  46. Daily market news sentiment and stock prices (2019) — link

1 of 54 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.