Paper: arXiv 2601.10143

Authors: Haochong Xia, Yao Long Teng, Regan Tan, Molei Qin, Xinrun Wang, Bo An

Abstract

In quantitative finance, the gap between training and real-world performance-driven by concept drift and distributional non-stationarity-remains a critical obstacle for building reliable data-driven systems. Models trained on static historical data often overfit, resulting in poor generalization in dynamic markets. The mantra “History Is Not Enough” underscores the need for adaptive data generation that learns to evolve with the market rather than relying solely on past observations. We present a drift-aware dataflow system that integrates machine learning-based adaptive control into the data curation process. The system couples a parameterized data manipulation module comprising single-stock transformations, multi-stock mix-ups, and curation operations, with an adaptive planner-scheduler that employs gradient-based bi-level optimization to control the system. This design unifies data augmentation, curriculum learning, and data workflow management under a single differentiable framework, enabling provenance-aware replay and continuous data quality monitoring. Extensive experiments on forecasting and reinforcement learning trading tasks demonstrate that our framework enhances model robustness and improves risk-adjusted returns. The system provides a generalizable approach to adaptive data management and learning-guided workflow automation for financial data.

Complexity vs Empirical Score

  • Math Complexity: 8.0/10
  • Empirical Rigor: 7.0/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper introduces a complex bi-level optimization framework and differentiable dataflow architecture with advanced ML concepts, scoring high on math. It is backed by extensive experiments on forecasting and RL trading tasks with reported improvements in risk-adjusted returns, demonstrating strong empirical validation.

Research Flowchart

  flowchart TD
  A["Research Goal:<br/>Adaptive Data System<br/>for Financial Time-Series"] --> B{"Input: Historical Data & Concept Drift"}
  B --> C["Methodology:<br/>Drift-Aware Dataflow System"]
  C --> D["Core: Parameterized<br/>Data Manipulation Module"]
  C --> E["Core: Adaptive Planner-Scheduler<br/>w/ Bi-Level Optimization"]
  D & E --> F["Process: Differentiable<br/>Data Augmentation & Curation"]
  F --> G["Output: Adaptive<br/>Synthetic Data Stream"]
  G --> H["Findings:<br/>Enhanced Model Robustness &<br/>Improved Risk-Adjusted Returns"]

Cited References (56)

  1. When to learn what: Model-adaptive data augmentation curriculum (2023)
  2. Finding order in chaos: A novel data augmentation method for time series in contrastive learning (2024)
  3. On LLMs-driven synthetic data generation, curation, and evaluation: A survey (2024) — link
  4. Covariate shift adaptation by importance weighted cross validation. (2007)
  5. Contrastive curriculum learning for sequential user behavior modeling via data augmentation (2021)
  6. DDG-DA: Data distribution generation for predictable concept drift adaptation (2022)
  7. Augmax: Adversarial composition of random augmentations for robust training (2021)
  8. An empirical survey of data augmentation for time series classification with neural networks (2021)
and 48 more
  1. Randaugment: Practical automated data augmentation with a reduced search space (2020)
  2. Neural stochastic agent-based limit order book simulation: A hybrid methodology — → 2303.00080
  3. Agent-based models of financial markets (2007)
  4. Limit order books (2013) — link
  5. Statistical analysis of financial returns for a multiagent order book model of asset trading (2007) — link
  6. Get real: Realism metrics for robust limit order book market simulations (2020)
  7. State dependent parallel neural Hawkes process for limit order book event stream prediction and simulation (2022)
  8. Modeling financial time-series with generative adversarial networks (2019)
  9. Autoaugment: Learning augmentation policies from data (2018) — link
  10. Adaaug: Learning class-and instance-adaptive data augmentation policies (2021)
  11. Adversarial autoaugment (2019) — link
  12. Curriculum learning (2009)
  13. Self-paced curriculum learning (2015)
  14. Market-GAN: Adding control to financial market data generation with semantic context (2024)
  15. Simple and effective curriculum pointer-generator networks for reading comprehension over long narratives (2019) — link
  16. FinAgent: A multimodal foundation agent for financial trading: Tool-augmented, diversified, and generalist — → 2402.18485
  17. Earnhft: Efficient hierarchical reinforcement learning for high frequency trading (2024)
  18. TradeMaster: A holistic quantitative trading platform empowered by reinforcement learning (2024)
  19. Stock price forecasting with deep learning: A comparative study (2020)
  20. Factorvae: A probabilistic dynamic factor model based on variational autoencoder for predicting cross-sectional stock returns (2022)
  21. An overview of machine learning, deep learning, and reinforcement learning-based techniques in quantitative finance: Recent progress and challenges (2023)
  22. Generalizing to unseen domains: A survey on domain generalization (2022)
  23. On uncertainty, tempering, and data augmentation in bayesian classification (2022)
  24. On-manifold adversarial data augmentation improves uncertainty calibration (2021)
  25. Aleatory and epistemic uncertainty in probability elicitation with an example from hazardous waste management (1996)
  26. Meta-weight-net: Learning an explicit mapping for sample weighting (2019)
  27. Metaaugment: Sample-aware data augmentation policy learning (2021)
  28. How to train your MAML (2019)
  29. Data parameters: A new family of parameters for learning a differentiable curriculum (2019)
  30. Fast autoaugment (2019)
  31. Visualizing data using t-SNE. (2008)
  32. MacroHFT: Memory augmented context-aware reinforcement learning On high frequency trading — → 2406.14537
  33. Trivialaugment: Tuning-free yet state-of-the-art data augmentation (2021)
  34. Informer: Beyond efficient transformer for long sequence time-series forecasting (2021)
  35. Population based augmentation: Efficient learning of augmentation policy schedules (2019)
  36. Estimating or propagating gradients through stochastic neurons for conditional computation (2013) — link
  37. Differentiable automatic data augmentation (2020)
  38. Human-level control through deep reinforcement learning (2015)
  39. Proximal policy optimization algorithms (2017) — link
  40. Real-valued (medical) time series generation with recurrent conditional gans (2017) — link
  41. Time-series generative adversarial networks (2019)
  42. Generative moment matching networks (2015) — link
  43. Conditional Sig-Wasserstein GANs for time series generation (2020) — link
  44. A novel virtual sample generation method based on a modified conditional Wasserstein GAN to address the small sample size problem in soft sensing (2022)
  45. CWGAN: Conditional wasserstein generative adversarial nets for fault data generation (2019)
  46. Diffusion-ts: Interpretable diffusion for general time series generation (2024) — link
  47. Data augmentation of wearable sensor data for parkinson’s disease monitoring using convolutional neural networks (2017)
  48. STL: A seasonal-trend decomposition (1990)

3 of 56 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.