Paper: arXiv 2510.15691

Authors: Tian Guo, Emmanuel Hauptmann

Abstract

In quantitative investing, return prediction supports various tasks, including stock selection, portfolio optimization, and risk management. Quantitative factors, such as valuation, quality, and growth, capture various characteristics of stocks. Unstructured data, like news and transcripts, has attracted growing attention, driven by recent advances in large language models (LLMs). This paper examines effective methods for leveraging multimodal factors and newsflow in return prediction and stock selection. First, we introduce a fusion learning framework to learn a unified representation from factors and newsflow representations generated by an LLM. Within this framework, we compare three methods of different architectural complexities: representation combination, representation summation, and attentive representations. Next, building on the limitation of fusion learning observed in empirical comparison, we explore the mixture model that adaptively combines predictions made by single modalities and their fusion. To mitigate the training instability of the mixture model, we introduce a decoupled training approach with theoretical insights. Finally, our experiments on real investment universes yield several insights into effective multimodal modeling of factors and news for stock return prediction and selection.

Complexity vs Empirical Score

  • Math Complexity: 5.5/10
  • Empirical Rigor: 6.0/10 (capped from 7.5: transaction costs not modelled; see penalty flags)
  • Quadrant: Street Traders — practical and empirical, lighter on theory

Why this score: The paper employs established mathematical concepts like multimodal fusion and mixture models but lacks heavy derivations, placing it in a moderate math complexity range. It is highly empirical, featuring experiments on real investment universes, backtest portfolio evaluations (long-only and long-short), and comparisons of model variants, indicating strong data and implementation focus.

Research Flowchart

  flowchart TD
  A["Research Goal: Compare & combine quantitative factors & LLM newsflow for stock return prediction"] --> B["Input Data: Quantitative Factors & LLM Newsflow Representations"]
  B --> C["Method 1: Fusion Learning Framework"]
  C --> D["Comparison: Representation Combination vs. Summation vs. Attentive"]
  C --> E["Finding 1: Fusion Learning Limitations"]
  E --> F["Method 2: Mixture Model (Adaptive Combination)"]
  F --> G["Constraint: Training Instability"]
  G --> H["Solution: Decoupled Training Approach"]
  H --> I["Outcome: Optimized Multimodal Stock Return Prediction & Selection"]

Cited References (158)

  1. Variational inference: A review for statisticians (2017)
  2. Advances in variational inference (2018)
  3. Convex optimization: Algorithms and complexity (2015)
  4. Optimization methods for large-scale machine learning (2018)
  5. A hierarchical multi-task approach for learning embeddings from semantic tasks (2019)
  6. Wider and deeper LLM networks are fairer LLM evaluators (2023) — link
  7. Spae: Semantic pyramid autoencoder for multimodal generation with frozen llms (2024)
  8. RWKV: Reinventing RNNs for the Transformer Era (2023) — link
and 150 more
  1. A survey of large language models (2023) — link
  2. Large language models in finance: A survey (2023)
  3. Attention is all you need (2017)
  4. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019)
  5. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (2019)
  6. Deberta: Decoding-enhanced bert with disentangled attention (2020) — link
  7. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing (2021) — link
  8. Text embeddings by weakly-supervised contrastive pre-training (2022) — link
  9. Improving text embeddings with large language models (2023) — link
  10. Multilingual e5 text embeddings: A technical report (2024) — link
  11. Llm2vec: Large language models are secretly powerful text encoders (2024) — link
  12. NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models (2024) — link
  13. Improving Language Understanding by Generative Pre-Training (2018)
  14. Llama 2: Open foundation and fine-tuned chat models (2023) — link
  15. Mistral 7B (2023) — link
  16. Transformers need glasses! Information over-squashing in language tasks (2024) — link
  17. The llama 3 herd of models (2024) — link
  18. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension (2019) — link
  19. Exploring the limits of transfer learning with a unified text-to-text transformer (2020)
  20. Sentence Bottleneck Autoencoders from Transformer Language Models (2021)
  21. SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval (2023)
  22. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters (2020)
  23. Supervised contrastive learning for pre-trained language model fine-tuning (2020) — link
  24. Lora: Low-rank adaptation of large language models (2021) — link
  25. Finetuned language models are zero-shot learners (2021) — link
  26. Parameter-efficient fine-tuning of large-scale pre-trained language models (2023)
  27. Dora: Weight-decomposed low-rank adaptation (2024) — link
  28. Scaling instruction-finetuned language models (2024)
  29. Language models are unsupervised multitask learners (2019)
  30. Language models are few-shot learners (2020)
  31. Large language models are zero-shot reasoners (2022)
  32. Chain-of-thought prompting elicits reasoning in large language models (2022)
  33. Hybrid neural networks for learning the trend in time series (2017)
  34. Bitcoin volatility forecasting with a glimpse into buy and sell orders (2018)
  35. Sensing Social Media Signals for Cryptocurrency News (2019)
  36. Esg2risk: A deep learning framework from esg news to stock volatility prediction (2020) — link
  37. Exploring interpretable LSTM neural networks over multi-variable data (2019)
  38. Temporal mixture ensemble models for probabilistic forecasting of intraday cryptocurrency volume (2021)
  39. Learning Mixture Structure on Multi-Source Time Series for Probabilistic Forecasting
  40. The benefits of socially responsible investing: An active manager’s perspective (2015)
  41. Can ESG add alpha? An analysis of ESG tilt and momentum strategies (2016)
  42. Impact of ESG factors on firm risk in Europe (2016)
  43. Every little helps? ESG news and stock market reaction (2019)
  44. Novel volatility forecasting using deep learning–Long Short Term Memory Recurrent Neural Networks (2019)
  45. Hierarchical Multi-Scale Gaussian Transformer for Stock Movement Prediction. (2020)
  46. Uncertainty-aware lookahead factor models for quantitative investing (2020)
  47. Factorvae: A probabilistic dynamic factor model based on variational autoencoder for predicting cross-sectional stock returns (2022)
  48. Efficient integration of multi-order dynamics and internal dynamics in stock movement prediction (2023)
  49. Multifactor explanations of asset pricing anomalies (1996)
  50. Foundations of factor investing (2013)
  51. The supraview of return predictive signals (2013)
  52. Asset management: A systematic approach to factor investing (2014)
  53. Comparing the effectiveness of deep feedforward neural networks and shallow architectures for predicting stock price indices (2020)
  54. Empirical asset pricing via machine learning (2020)
  55. Deep learning in asset pricing (2024)
  56. Applications of deep learning in stock market prediction: recent progress (2021)
  57. Fine-Tuning Large Language Models for Stock Return Prediction Using Newsflow — → 2407.18103
  58. Modeling News Interactions and Influence for Financial Market Prediction — → 2410.10614
  59. On Assessing the Performance of LLMs for Target-Level Sentiment Analysis in Financial News Headlines (2025)
  60. Assessing consistency and reproducibility in the outputs of large language models: Evidence across diverse finance and accounting tasks — → 2503.16974
  61. Caution ahead: Numerical reasoning and look-ahead bias in AI models (2024)
  62. The Memorization Problem: Can We Trust LLMs’ Economic Forecasts? — → 2504.14765
  63. Entity Neutering (2025)
  64. Daily market news sentiment and stock prices (2019)
  65. Measuring news sentiment (2022)
  66. Vader: A parsimonious rule-based model for sentiment analysis of social media text (2014)
  67. FinVADER: Financial Sentiment Analysis (2023) — link
  68. FinBERT: Financial Sentiment Analysis with Pre-trained Language Models (2019) — link
  69. Finbert: A pretrained language model for financial communications (2020) — link
  70. Finbert: A pre-trained financial language representation model for financial text mining (2021)
  71. FinBERT-FOMC: Fine-Tuned FinBERT Model with sentiment focus method for enhancing sentiment analysis of FOMC minutes (2023)
  72. Leveraging BERT to Improve the FEARS Index for Stock Forecasting (2019)
  73. HTML: Hierarchical Transformer-based Multi-task Learning for Volatility Prediction (2020)
  74. Group, Extract and Aggregate: Summarizing a Large Amount of Finance News for Forex Movement Prediction (2019) — link
  75. Hierarchical complementary attention network for predicting stock price movements with news (2018)
  76. Textual analysis of stock market prediction using breaking financial news: The AZFin text system (2009)
  77. Quantifying the relationship between financial news and the stock market (2013)
  78. EAN: Event Attention Network for Stock Price Trend Prediction based on Sentimental Embedding (2019)
  79. Predicting short-term stock prices using ensemble methods and online data sources (2018)
  80. Listening to chaotic whispers: A deep learning framework for news-oriented stock trend prediction (2018)
  81. What You Say and How You Say It Matters: Predicting Financial Risk Using Verbal and Vocal Cues (2019)
  82. Deep attentive learning for stock movement prediction from social media text and company correlations (2020)
  83. Ask" who", not" what": Bitcoin volatility forecasting with twitter data (2023)
  84. The Econometrics of Financial Markets (1997)
  85. When is a word “negative”? Financial analysts’ journal special issue on textual analysis (2011)
  86. Twitter mood predicts the stock market (2011)
  87. Stock market prediction via deep learning techniques: A survey — → 2212.12717
  88. FinTral: A Family of GPT-4 Level Multimodal Financial Large Language Models (2024)
  89. Open-finllms: Open multimodal large language models for financial applications — → 2408.11878
  90. A multimodal foundation agent for financial trading: Tool-augmented, diversified, and generalist (2024)
  91. Multimodal machine learning: A survey and taxonomy (2018)
  92. Multimodal deep learning (2011)
  93. Deep multimodal learning: A survey (2019)
  94. What makes multi-modal learning better than single (provably) (2021)
  95. Affective interaction: Attentive representation learning for multi-modal sentiment classification (2022)
  96. Deep multimodal data fusion (2024)
  97. A survey of multimodal learning: Methods, applications, and future (2025)
  98. Anymal: An efficient and scalable any-modality augmented language model (2024)
  99. Uniter: Universal image-text representation learning (2020)
  100. Multimodal transformer for unaligned multimodal language sequences (2019)
  101. Revisiting deep learning models for tabular data (2021)
  102. CAMEF: Causal-Augmented Multi-Modality Event-Driven Financial Forecasting by Integrating Time Series Patterns and Salient Macroeconomic Announcements (2025)
  103. Efficient large-scale multi-modal classification (2018)
  104. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework (2022)
  105. Attention bottlenecks for multimodal fusion (2021)
  106. Stock movement prediction from tweets and historical prices (2018)
  107. Financial forecasting from textual and tabular time series (2024)
  108. Guided attention multimodal multitask financial forecasting with inter-company relationships and global and local news (2022)
  109. CausalStock: Deep end-to-end causal discovery for news-driven multi-stock movement prediction (2024)
  110. Expected returns and large language models (2022)
  111. Re(Visiting) Large Language Models in Finance (2024)
  112. Understanding llm embeddings for regression (2024) — link
  113. OmniPred: Language Models as Universal Regressors (2024)
  114. Can chatgpt forecast stock price movements? return predictability and large language models — → 2304.07619
  115. Harnessing LLMs for temporal data-a study on explainable financial time series forecasting (2023)
  116. LLMFactor: Extracting Profitable Factors through Prompts for Explainable Stock Movement Prediction (2024) — link
  117. Financial statement analysis with large language models — → 2407.17866
  118. Ploutos: Towards interpretable stock movement prediction with financial large language model — → 2403.00782
  119. Learning to generate explainable stock predictions using self-reflective large language models — → 2402.03659
  120. DeFine: Decision-Making with Analogical Reasoning over Factor Profiles (2025)
  121. Finllama: Llm-based financial sentiment analysis for algorithmic trading (2024)
  122. AlphaFin: Benchmarking Financial Analysis with Retrieval-Augmented Stock-Chain Framework (2024)
  123. Twenty years of mixture of experts (2012)
  124. Variational mixture-of-experts autoencoders for multi-modal deep generative models (2019)
  125. Filtered not Mixed: Filtering-Based Online Gating for Mixture of Large Language Models (2025)
  126. DHMoE: Diffusion Generated Hierarchical Multi-Granular Expertise for Stock Prediction (2025)
  127. Mastering stock markets with efficient mixture of diversified trading experts (2023)
  128. Tradexpert: Revolutionizing trading with mixture of expert llms — → 2411.00782
  129. Stock movement prediction with financial news using contextualized embedding from bert (2021) — link
  130. Pytorch: An imperative style, high-performance deep learning library (2019)
  131. Huggingface’s transformers: State-of-the-art natural language processing (2019) — link
  132. Mitigating modality collapse in multimodal VAEs via impartial optimization (2022)
  133. A Closer Look at Multimodal Representation Collapse (2022)
  134. Machine learning and the cross-section of emerging market stock returns (2023)
  135. Beating the market with a bad predictive model (2023)
  136. Machine learning models predicting returns: Why most popular performance metrics are misleading and proposal for an efficient metric (2022)
  137. Decision-informed neural networks with large language model integration for portfolio optimization — → 2502.00828
  138. Information Coefficient as a Performance Measure of Stock Selection Models (2020) — link
  139. Is All the Information in the Price? LLM Embeddings versus the EMH in Stock Clustering — → 2509.01590
  140. The adaptive markets hypothesis: Market efficiency from an evolutionary perspective (2005)
  141. The Fama-French five-factor model and emerging market equity returns (2022)
  142. Cross-country determinants of market efficiency: a technical analysis perspective (2024)
  143. Collective dynamics of stock market efficiency (2020)
  144. How news and its context drive risk and returns around the world (2019)
  145. Mixing It Up: The Cocktail Effect of Multi-Task Fine-Tuning on LLM Performance–A Case Study in Finance (2024) — link
  146. PIXIU: a large language model, instruction data and evaluation benchmark for finance (2023)
  147. jina-embeddings-v3: Multilingual embeddings with task lora (2024) — link
  148. Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models (2025) — link
  149. Demystifying domain-adaptive post-training for financial llms (2025) — link
  150. A Dynamic Regime-Switching Model Using Gated Recurrent Straight-Through Units. (2024)

13 of 158 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.