Paper: arXiv 2403.08362

Abstract

Microcanonical gradient descent is a sampling procedure for energy-based models allowing for efficient sampling of distributions in high dimension. It works by transporting samples from a high-entropy distribution, such as Gaussian white noise, to a low-energy region using gradient descent. We put this model in the framework of normalizing flows, showing how it can often overfit by losing an unnecessary amount of entropy in the descent. As a remedy, we propose a mean-field microcanonical gradient descent that samples several weakly coupled data points simultaneously, allowing for better control of the entropy loss while paying little in terms of likelihood fit. We study these models in the context of financial time series, illustrating the improvements on both synthetic and real data.

Complexity vs Empirical Score

  • Math Complexity: 8.0/10
  • Empirical Rigor: 7.0/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper introduces and analyzes a novel generative model (MF-MGDM) with substantial theoretical derivation and entropy bounds, indicating high mathematical complexity, while also evaluating the method on synthetic and real financial data, demonstrating practical implementation and validation.

Research Flowchart

  flowchart TD
  A["Research Goal<br>Efficient Sampling for EBM"] --> B{"Methodology<br>Microcanonical Gradient Descent"}
  B --> C["Input: High-Entropy Data<br>Gaussian White Noise"]
  C --> D["Process: Mean-Field Optimization<br>Simultaneous weakly coupled descent"]
  D --> E["Output: Low-Energy Distribution<br>Sampled Time Series"]
  E --> F["Key Findings"]
  subgraph F ["Outcomes"]
      F1["Better entropy control<br>Less loss than normalizing flows"]
      F2["Efficient likelihood fit<br>in high dimensions"]
      F3["Effective on Financial Data<br>Synthetic & Real"]
  end

Cited References (140)

  1. Information Theory and Statistical Mechanics (1957)
  2. Maximum-entropy distributions having prescribed first and second moments (1973)
  3. Time evolution of large classical systems (1975)
  4. An equilibrium characterization of the term structure — link
  5. A Theory of the Term Structure of Interest Rates (1985) — link
  6. Large deviations for stationary Gaussian processes — link
  7. Modern spectral estimation
  8. Matching pursuits with time-frequency dictionaries (1993) — link
and 132 more
  1. A yield-factor model of interest rates — link
  2. Evaluating Density Forecasts with Applications to Financial Risk Management — link
  3. Gradient-Based Optimization of Hyperparameters
  4. Specification Analysis of Affine Term Structure Models — link
  5. Time Series and Cross-section Information in Affine Term-Structure Models — link
  6. Testing Density Forecasts, With Applications to Risk Management — link
  7. Term Premia and Interest Rate Forecasts in Affine Models — link
  8. Term Structure Dynamics in Theory and Reality — link
  9. Energy-Based Models for Sparse Overcomplete Representations
  10. Energy-based models for sparse overcomplete representations (2003)
  11. CAViaR: Conditional Autoregressive Value at Risk by Regression Quantiles — link
  12. The Statistical and Economic Role of Jumps in Continuous-Time Interest Rate Models — link
  13. Elements of Information Theory (2006)
  14. Forecasting the term structure of government bond yields — link
  15. Efficient Learning of Sparse Representationswith an Energy-Based Model — link
  16. Regime Shifts in a Dynamic Term Structure Model of U.S. Treasury Bond Yields — link
  17. Evidence on Simulation Inference for Near Unit-Root Processes with Implications for Term Structure Estimation — link
  18. Strictly Proper Scoring Rules, Prediction, and Estimation — link
  19. Global yield curve dynamics and interactions: A dynamic Nelson \textendash Siegel approach — link
  20. Multiscale Scattering for Audio Classification
  21. Forecasting with the term structure: The role of no-arbitrage restrictions (2011) — link
  22. Making and Evaluating Point Forecasts — link
  23. Comparing Density Forecasts Using Threshold- and Quantile-Weighted Scoring Rules — link
  24. A New Perspective on Gaussian Dynamic Term Structure Models — link
  25. Bayesian Time Series Models
  26. Group Invariant Scattering (2012)
  27. Approximate Bayesian Computational Methods — link
  28. Auto-Encoding Variational Bayes — link
  29. Learning from the past, predicting the statistics for the future, learning an evolving system — link
  30. Deep Scattering Spectrum (2014) — link
  31. Estimate nothing — link
  32. Rough paths, Signatures and the modelling of functions on streams (2014) — link
  33. Doubly Stochastic Variational Bayes for non-Conjugate Inference
  34. Coherence and Elicitability — link
  35. Gradient-based Hyperparameter Optimization through Reversible Learning
  36. Variational Inference with Normalizing Flows — link
  37. A note on the evaluation of generative models — link
  38. Higher order elicitability and Osband’s principle — link
  39. Understanding deep convolutional networks (2016)
  40. WaveNet: A Generative Model for Raw Audio — link
  41. Approximate maximum likelihood estimation using data-cloning ABC
  42. Forecasting of stock return prices with sparse representation of financial time series over redundant dictionaries — link
  43. Matching Networks for One Shot Learning
  44. Solving ill-posed inverse problems using iterative deep neural networks — link
  45. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
  46. Prototypical Networks for Few-shot Learning
  47. Staying on Top of the Curve: A Cascade Model of Term Structure Dynamics — link
  48. Quantile-Based Risk Sharing — link
  49. Robustness in the Optimization of Risk Measures — link
  50. Bilevel Programming for Hyperparameter Optimization and Meta-Learning
  51. Learning deep representations by mutual information estimation and maximization — link
  52. A Neural Stochastic Volatility Model — link
  53. Concept Learning with Energy-Based Models — link
  54. Representation Learning with Contrastive Predictive Coding — link
  55. Meta-Learning with Differentiable Closed-Form Solvers
  56. Multiscale sparse microcanonical models (2019)
  57. Multiscale Sparse Microcanonical Models (2019)
  58. Image Deformation Meta-Networks for One-Shot Learning
  59. Relaxing Bijectivity Constraints with Continuously Indexed Normalising Flows (2019) — link
  60. Domain Adaption in One-Shot Learning
  61. Implicit Generation and Generalization in Energy-Based Models (2019) — link
  62. (Martingale) Optimal Transport and Anomaly Detection with Neural Networks: A Primal-Dual Algorithm — link
  63. Meta-Learning Representations for Continual Learning — link
  64. Maximum Entropy Generators for Energy-Based Models — link
  65. Meta-Learning With Differentiable Convex Optimization
  66. Maximum-entropy Scattering Models for Financial Time Series (2019)
  67. Phase harmonic correlations and convolutional neural networks (2019)
  68. Normalizing Flows for Probabilistic Modeling and Inference (2021) — link
  69. Meta-Learning with Implicit Gradients
  70. Sliced Score Matching: A Scalable Approach to Density and Score Estimation — link
  71. Learning Likelihoods with Conditional Normalizing Flows — link
  72. Kymatio: Scattering Transforms in Python (2020)
  73. A Data-Driven Market Simulator for Small Data Environments (2020) — link
  74. A new approach to observational cosmology using the scattering transform — link
  75. Compositional Visual Generation and Inference with Energy Based Models — link
  76. Implicit Generation and Modeling with Energy-Based Models — link
  77. Quantile-Based Risk Sharing with Heterogeneous Beliefs — link
  78. Deep Declarative Networks: A New Hope
  79. Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One — link
  80. Learning the Stein Discrepancy for Training and Evaluating Energy-Based Models without Sampling — link
  81. Why Normalizing Flows Fail to Detect Out-of-Distribution Data
  82. Data Anonymisation, Outlier Detection and Fighting Overfitting with Restricted Boltzmann Machines — link
  83. Optimizing Millions of Hyperparameters by Implicit Differentiation
  84. Conditional Sig-Wasserstein GANs for Time Series Generation — link
  85. SurVAE Flows: Surjections to Bridge the Gap between VAEs and Flows — link
  86. Generalizing from a Few Examples: A Survey on Few-shot Learning — link
  87. Quant GANs: deep generation of financial time series — link
  88. Black-box model risk in finance — link
  89. Unsupervised Learning of Compositional Energy Concepts — link
  90. Bayes risk, elicitability, and the Expected Shortfall — link
  91. Learning Energy-Based Models by Diffusion Recovery Likelihood — link
  92. Tackling the exponential scaling of signature-based GANs for high-dimensional financial time series generation — link
  93. Penalty Method for Inversion-Free Deep Bilevel Optimization
  94. Sig- W asserstein GAN s for Time Series Generation (2021) — link
  95. How to Train Your Energy-Based Models — link
  96. A Preferred-Habitat Model of the Term Structure of Interest Rates — link
  97. An Axiomatic Foundation for the Expected Shortfall
  98. Multi-Asset Spot and Option Market Simulation — link
  99. Bilevel Programs Meet Deep Learning: A Unifying View on Inference Learning Methods
  100. Maximum entropy models from phase harmonic covariances (2021)
  101. Generalized Rectifier Wavelet Covariance Models For Texture Synthesis (2022)
  102. Energy-Based Models for Continual Learning — link
  103. TTS-GAN: A Transformer-based Time-Series Generative Adversarial Network
  104. Score Matched Neural Exponential Families for Likelihood-Free Inference — link
  105. Separation of dust emission from the cosmic infrared background in Herschel observations with wavelet phase harmonics (2023)
  106. Scattering Spectra Models for Physics (2023) — link
  107. One-shot learning of stochastic differential equations with dataadapted kernels — link
  108. Bridging mean-field games and normalizing flows with trajectory regularization — link
  109. Path Shadowing Monte-Carlo — link
  110. Scale Dependencies and Self-Similar Models with Wavelet Scattering Spectra (2023)
  111. bashtage/arch: Release 6.2 (2023) — link
  112. ()
  113. Probabilistic theory of mean field games with applications I-II (2018)
  114. Gradient Descent Only Converges to Minimizers: Non-Isolated Critical Points and Invariant Regions (2016) — link
  115. Generative modeling by estimating gradients of the data distribution (2019)
  116. Probabilistic inference using Markov chain Monte Carlo methods (1993)
  117. Invertible residual networks (2019)
  118. Diffusion models: A comprehensive survey of methods and applications (2023)
  119. Score-Based Generative Modeling through Stochastic Differential Equations (2020)
  120. Monte Carlo statistical methods (1999)
  121. Interacting particle solutions of fokker–planck equations through gradient–log–density estimation (2020)
  122. Large Deviation Principles and Complete Equivalence and Nonequivalence Results for Pure and Mixed Ensembles (2000)
  123. Equivalence and Nonequivalence of Ensembles: Thermodynamic, Macrostate, and Measure Levels (2015)
  124. Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images (1984)
  125. Invariant Scattering Convolution Networks (2013)
  126. Scattering Networks for Hybrid Representation Learning (2019)
  127. Joint Time–Frequency Scattering (2019)
  128. Rotation, Scaling and Deformation Invariant Scattering for Texture Discrimination (2013)
  129. Solid Harmonic Wavelet Scattering: Predicting Quantum Molecular Energy from Invariant Descriptors of 3D Electronic Densities (2017)
  130. Arrhythmia Classification of 12-lead Electrocardiograms by Hybrid Scattering- LSTM Networks (2020)
  131. Time–frequency scattering accurately models auditory similarities between instrumental playing techniques (2021)
  132. PyTorch: An Imperative Style, High-Performance Deep Learning Library (2019)

0 of 140 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.