Paper: arXiv 2412.10540

Abstract

In this paper, we tackle the challenge of predicting stock movements in financial markets by introducing Higher Order Transformers, a novel architecture designed for processing multivariate time-series data. We extend the self-attention mechanism and the transformer architecture to a higher order, effectively capturing complex market dynamics across time and variables. To manage computational complexity, we propose a low-rank approximation of the potentially large attention tensor using tensor decomposition and employ kernel attention, reducing complexity to linear with respect to the data size. Additionally, we present an encoder-decoder model that integrates technical and fundamental analysis, utilizing multimodal signals from historical prices and related tweets. Our experiments on the Stocknet dataset demonstrate the effectiveness of our method, highlighting its potential for enhancing stock movement prediction in financial markets.

Complexity vs Empirical Score

  • Math Complexity: 8.5/10
  • Empirical Rigor: 5.5/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper introduces a novel, mathematically complex higher-order transformer architecture using tensor decomposition and kernel attention, with rigorous theoretical formulations. It provides empirical validation on a specific dataset (Stocknet) with multimodal data (prices and tweets), though the summary lacks details on code, backtesting specifics, or extensive statistical metrics, making it more research-oriented than immediately backtest-ready.

Research Flowchart

  flowchart TD
  A["Research Goal: Enhance Stock Movement Prediction<br>on Multimodal Time-Series Data"] --> B["Data/Inputs"]
  B --> B1["Historical Prices<br>Technical Analysis"]
  B --> B2["Related Tweets<br>Fundamental Analysis"]
  B1 & B2 --> C["Methodology: Higher Order Transformers"]
  C --> D["Computational Processes"]
  D --> D1["Extend Self-Attention to Higher Order"]
  D --> D2["Low-Rank Tensor Decomposition<br>+ Kernel Attention<br>Complexity: Linear O(N)"]
  D1 & D2 --> E["Key Findings/Outcomes"]
  E --> E1["Effective Capture of Complex<br>Market Dynamics"]
  E --> E2["Enhanced Prediction Accuracy<br>on Stocknet Dataset"]

Cited References (39)

  1. Predicting the oil prices: do technical indicators help? (2016)
  2. Forecasting directional movement of Forex data using LSTM with technical and macroeconomic indicators (2021)
  3. Generalized autoregressive conditional heteroskedasticity (1986)
  4. Stock trend prediction using news sentiment analysis (2016) — link
  5. Fundamental analysis, future earnings, and stock prices (1997)
  6. Listening to chaotic whispers: A deep learning framework for news-oriented stock trend prediction (2018)
  7. Deep learning for event-driven stock prediction (2015)
  8. Stock movement prediction from tweets and historical prices (2018)
and 31 more
  1. Temporal relational ranking for stock prediction (2019)
  2. Spatiotemporal hypergraph convolution network for stock movement forecasting (2020)
  3. Hats: A hierarchical graph attention network for stock movement prediction (2019) — link
  4. Exploring graph neural networks for stock market predictions with rolling window analysis (2019) — link
  5. FinKG: A Core Financial Knowledge Graph for Financial Analysis (2023)
  6. Stock movement prediction and portfolio management via multimodal learning with transformer (2021)
  7. A multimodal event-driven LSTM model for stock prediction using online news (2020)
  8. Deep attentive learning for stock movement prediction from social media text and company correlations (2020)
  9. Transformers Generalize DeepSets and Can be Extended to Graphs and Hypergraphs (2022) — link
  10. Rethinking Attention with Performers (2022) — link
  11. RoFormer: Enhanced Transformer with Rotary Position Embedding (2023) — link
  12. Listening to Chaotic Whispers: A Deep Learning Framework for News-oriented Stock Trend Prediction (2019) — link
  13. Fundamental Analysis based Neural Network for Stock Movement Prediction (2022) — link
  14. LLaMA: Open and Efficient Foundation Language Models (2023) — link
  15. Root Mean Square Layer Normalization (2019) — link
  16. The Wall Street Neophyte: A Zero-Shot Analysis of ChatGPT Over MultiModal Stock Movement Prediction Challenges (2023) — link
  17. FinBERT: Financial Sentiment Analysis with Pre-trained Language Models (2019) — link
  18. Lag-Llama: Towards Foundation Models for Probabilistic Time Series Forecasting (2024) — link
  19. Vision Transformers Need Registers (2024) — link
  20. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019) — link
  21. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2021) — link
  22. Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting (2021) — link
  23. Attention Is All You Need (2023) — link
  24. Smoothing, forecasting and prediction of discrete time series (2004)
  25. Astock: A New Dataset and Automated Stock Trading based on Stock-specific News Analyzing Model (2022) — link
  26. Transformer-Based Deep Learning Model for Stock Price Prediction: A Case Study on Bangladesh Stock Market (2023) — link
  27. Transformer-based attention network for stock movement prediction (2022) — link
  28. Transformer-Based Capsule Network For Stock Movement Prediction (2019) — link
  29. UniT: Multimodal Multitask Learning with a Unified Transformer (2021) — link
  30. Multimodal Transformer for Unaligned Multimodal Language Sequences (2019) — link
  31. Accurate Stock Movement Prediction with Self-supervised Learning from Sparse Noisy Tweets (2022) — link

0 of 39 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.