Is attention truly all we need? An empirical study of asset pricing in pretrained RNN sparse and global attention models

Paper: arXiv 2508.19006

Authors: Shanyan Lai

Abstract

This study investigates the pretrained RNN attention models with the mainstream attention mechanisms such as additive attention, Luong’s three attentions, global self-attention (Self-att) and sliding window sparse attention (Sparse-att) for the empirical asset pricing research on top 420 large-cap US stocks. This is the first paper on the large-scale state-of-the-art (SOTA) attention mechanisms applied in the asset pricing context. They overcome the limitations of the traditional machine learning (ML) based asset pricing, such as mis-capturing the temporal dependency and short memory. Moreover, the enforced causal masks in the attention mechanisms address the future data leaking issue ignored by the more advanced attention-based models, such as the classic Transformer. The proposed attention models also consider the temporal sparsity characteristic of asset pricing data and mitigate potential overfitting issues by deploying the simplified model structures. This provides some insights for future empirical economic research. All models are examined in three periods, which cover pre-COVID-19 (mild uptrend), COVID-19 (steep uptrend with a large drawdown) and one year post-COVID-19 (sideways movement with high fluctuations), for testing the stability of these models under extreme market conditions. The study finds that in value-weighted portfolio back testing, Model Self-att and Model Sparse-att exhibit great capabilities in deriving the absolute returns and hedging downside risks, while they achieve an annualized Sortino ratio of 2.0 and 1.80 respectively in the period with COVID-19. And Model Sparse-att performs more stably than Model Self-att from the perspective of absolute portfolio returns with respect to the size of stocks’ market capitalization.

Complexity vs Empirical Score

• Math Complexity: 7.5/10
• Empirical Rigor: 9.0/10
• Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper employs advanced mathematical concepts including RNN variants, attention mechanisms (global self-attention, sliding window sparse attention), and causal masking, requiring dense theoretical exposition, but it is heavily empirical with a large-scale study on 420 stocks, multiple market regimes, specific backtesting metrics like Sortino ratios, and rigorous stability testing.

Research Flowchart

  flowchart TD
  A["Research Goal<br>Compare SOTA Attention Models<br>in Empirical Asset Pricing"] --> B["Data & Model Setup<br>Top 420 US Large-Cap Stocks<br>3 Market Periods<br>Causal Masks Applied"]
  B --> C["Model Architecture<br>Pretrained RNN + Attention<br>Additive | Luong<br>Global Self-Attention<br>Sliding Window Sparse-Attention"]
  C --> D["Computational Process<br>Train Models &<br>Value-Weighted Portfolio Backtesting"]
  D --> E["Key Findings<br>Self-att & Sparse-att:<br>High Returns & Downside Risk Hedging<br>Sparse-att: Best Stability<br>by Market Cap Size"]

Cited References (716)

1. An Augmented q -Factor Model with Expected Growth* (2021) — link
2. 1 Econometric evaluation of asset pricing models (1996) — link
3. A census of the factor zoo (2019)
4. A closer look at the short-term return reversal (2014) — link
5. A comparative analysis of lstm versus patchtst in predictive modeling of asset prices (2024)
6. A comprehensive look at the empirical performance of equity premium prediction (2008)
7. A conditional fuzzy inference approach in forecasting (2020) — link
8. A cross-sectional test of an investment-based asset pricing model (1996)

15 of 716 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.