Paper: arXiv 2502.02199

Abstract

Large language models (LLMs) have shown remarkable success in language modelling due to scaling laws found in model size and the hidden dimension of the model’s text representation. Yet, we demonstrate that compressed representations of text can yield better performance in LLM-based regression tasks. In this paper, we compare the relative performance of embedding compression in three different signal-to-noise contexts: financial return prediction, writing quality assessment and review scoring. Our results show that compressing embeddings, in a minimally supervised manner using an autoencoder’s hidden representation, can mitigate overfitting and improve performance on noisy tasks, such as financial return prediction; but that compression reduces performance on tasks that have high causal dependencies between the input and target data. Our results suggest that the success of interpretable compressed representations such as sentiment may be due to a regularising effect.

Complexity vs Empirical Score

  • Math Complexity: 4.0/10
  • Empirical Rigor: 7.5/10
  • Quadrant: Street Traders — practical and empirical, lighter on theory

Why this score: The paper employs standard machine learning techniques (autoencoders, regression) with minimal advanced mathematical derivations, but demonstrates strong empirical rigor through the use of three distinct real-world datasets (including financial news and stock returns), specific model architectures (all-mpnet-base-v2), and clear performance benchmarking, positioning it as a practical, data-heavy investigation.

Research Flowchart

  flowchart TD
  A["Research Goal<br/>Does LLM embedding compression<br/>improve regression performance?"]
  B["Dataset & Signal-to-Noise Contexts<br/>Financial (Low SNR)<br/>Writing Quality (High SNR)<br/>Review Scoring (Medium SNR)"]
  C["Methodology<br/>Train Autoencoder on LLM Embeddings<br/>Compress to latent space representation"]
  D["Computation<br/>Regression Task: Predict Target<br/>Compare Raw vs. Compressed Embeddings"]
  E{"Key Findings & Outcomes"}
  E --> F["Low SNR (Financial)<br/>Compression improves performance<br/>Reduces overfitting"]
  E --> G["High SNR (Writing/Review)<br/>Compression reduces performance<br/>Removes causal dependencies"]
  E --> H["Interpretability<br/>Compressed features act as<br/>effective regularization"]

  A --> B
  B --> C
  C --> D
  D --> E

Cited References (56)

  1. The Theory of Parsing, Translation and Compiling (1972)
  2. Dissociating Language and Thought in Large Language Models (2024) — link
  3. Re (Visiting) Large Language Models in Finance (2024) — link
  4. Graph-enhanced Large Language Models in Asynchronous Plan Reasoning (2025) — link
  5. Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistency (2024) — link
  6. Age Related Differences in Language Usage and Reading between E nglish Monolinguals and Bilinguals (2013) — link
  7. Challenges and Strategies in Cross-Cultural NLP (2022) — link
  8. Geographic Adaptation of Pretrained Language Models (2024) — link
and 48 more
  1. Bridging Language and Items for Retrieval and Recommendation (2024) — link
  2. Feedback Prize - English Language Learning (2022) — link
  3. Software Applications User Reviews (2017) — link
  4. Efficient Prompting Methods for Large Language Models: A Survey (2024) — link
  5. Predicting Stock Returns in the Presence of Uncertain Structural Changes and Sample Noise (2017) — link
  6. LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders (2024) — link
  7. Macroeconomic Forecasting Through News, Emotions and Narrative (2021) — link
  8. Automatic Detection of Relevant Information, Predictions and Forecasts in Financial News Through Topic Modelling with Latent Dirichlet Allocation (2023) — link
  9. Are LLMs Capable of Data-based Statistical and Causal Reasoning? Benchmarking Advanced Quantitative Reasoning with Data (2024) — link
  10. Exploring the Numerical Reasoning Capabilities of Language Models: A Comprehensive Analysis on Tabular Data (2023) — link
  11. AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts (2022) — link
  12. Training Compute-optimal Large Language Models (2022) — link
  13. Regression or Classification? Automated Essay Scoring for N orwegian (2019) — link
  14. The Probable Error of a Mean (1908) — link
  15. A Comprehensive Survey on Regularization Strategies in Machine Learning (2022) — link
  16. To Repeat or Not To Repeat: Insights from Scaling LLM Under Token-crisis (2023) — link
  17. Random Forests (2001) — link
  18. Robustness of Random Forests for Regression (2012) — link
  19. news-please: A Generic News Crawler and Extractor (2017) — link
  20. Emotion English DistilRoBERTa-base (2022)
  21. DistilBERT: A Distilled Version of BERT—Smaller, Faster, Cheaper, and Lighter (2019) — link
  22. Understanding LLM Embeddings for Regression (2024) — link
  23. NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models (2024) — link
  24. Are Decoder-Only Language Models Better than Encoder-Only Language Models in Understanding Word Meaning? (2024) — link
  25. Noise (1986) — link
  26. Is All That Talk Just Noise? The Information Content of Internet Stock Message Boards (2004) — link
  27. Deep Attentive Learning for Stock Movement Prediction From Social Media Text and Company Correlations (2020) — link
  28. Forecasting COVID -19 Caseloads Using Unsupervised Embedding Clusters of Social Media Posts (2022) — link
  29. Scaling Laws for Neural Language Models (2020) — link
  30. Sentence- BERT : Sentence Embeddings using S iamese BERT -Networks (2019) — link
  31. MPNet: Masked and Permuted Pre-training for Language Understanding (2020) — link
  32. MTEB: Massive Text Embedding Benchmark (2023) — link
  33. Forecasting Credit Ratings: A Case Study where Traditional Methods Outperform Generative LLMs (2025) — link
  34. The Llama 3 Herd of Models (2024) — link
  35. Introduction to the C o NLL -2003 Shared Task: Language-Independent Named Entity Recognition (2003) — link
  36. BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding (2019) — link
  37. From Amateurs to Connoisseurs: Modeling the Evolution of User Expertise Through Online Reviews (2013) — link
  38. Robust Estimation of a Location Parameter (1964) — link
  39. Principal Component Analysis (1987) — link
  40. UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction (2018) — link
  41. Visualizing Data using t-SNE (2008) — link
  42. Robust Estimation of a Location Parameter (1992) — link
  43. Publications Manual (1983)
  44. Character-level Convolutional Networks for Text Classification (2015) — link
  45. Alternation (1981) — link
  46. Scalable training of L1 -regularized log-linear models (2007)
  47. Algorithms on Strings, Trees and Sequences (1997)
  48. Yara Parser: A Fast and Accurate Dependency Parser (2015) — link

0 of 56 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.