End-to-End Large Portfolio Optimization for Variance Minimization with Neural Networks through Covariance Cleaning

Paper: arXiv 2507.01918

Authors: Christian Bongiorno, Efstratios Manolakis, Rosario Nunzio Mantegna

Abstract

We develop a rotation-invariant neural network that provides the global minimum-variance portfolio by jointly learning how to lag-transform historical returns and how to regularise both the eigenvalues and the marginal volatilities of large equity covariance matrices. This explicit mathematical mapping offers clear interpretability of each module’s role, so the model cannot be regarded as a pure black-box. The architecture mirrors the analytical form of the global minimum-variance solution yet remains agnostic to dimension, so a single model can be calibrated on panels of a few hundred stocks and applied, without retraining, to one thousand US equities-a cross-sectional jump that demonstrates robust out-of-sample generalisation. The loss function is the future realized minimum portfolio variance and is optimized end-to-end on real daily returns. In out-of-sample tests from January 2000 to December 2024 the estimator delivers systematically lower realised volatility, smaller maximum drawdowns, and higher Sharpe ratios than the best analytical competitors, including state-of-the-art non-linear shrinkage. Furthermore, although the model is trained end-to-end to produce an unconstrained (long-short) minimum-variance portfolio, we show that its learned covariance representation can be used in general optimizers under long-only constraints with virtually no loss in its performance advantage over competing estimators. These gains persist when the strategy is executed under a highly realistic implementation framework that models market orders at the auctions, empirical slippage, exchange fees, and financing charges for leverage, and they remain stable during episodes of acute market stress.

Complexity vs Empirical Score

• Math Complexity: 8.5/10
• Empirical Rigor: 8.0/10
• Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper involves advanced mathematical concepts like rotation-invariant neural networks, eigenvalue regularization, and covariance matrix estimation, indicating high complexity. It also demonstrates strong empirical rigor with a 24-year out-of-sample backtest, realistic implementation constraints, and consistent outperformance against state-of-the-art benchmarks.

Research Flowchart

  flowchart TD
A["Research Goal:<br>Develop interpretable NN for<br>global minimum-variance portfolio<br>in large equity universes"] --> B["Data: Daily equity returns<br>Jan 2000 - Dec 2024"]
B --> C["Methodology:<br>Rotation-invariant NN<br>Lag-transformation +<br>Regularization of Eigenvalues &<br>Margin Vols"]
C --> D["Process:<br>End-to-end optimization<br>Loss = Realized future<br>minimum portfolio variance"]
D --> E["Outcomes:<br>Lower realized volatility,<br>smaller drawdowns,<br>higher Sharpe vs.<br>state-of-the-art competitors"]
E --> F["Robustness:<br>Works under long-only<br>constraints + realistic<br>trading costs"]

Cited References (112)

1. An Overview of Machine Learning for Portfolio Optimization. (2024)
2. Estimation of a covariance matrix (1975)
3. Portofolio selection (1952)
4. Cleaning large correlation matrices: tools from random matrix theory (2017)
5. Covariance matrix filtering and portfolio optimisation: the Average Oracle vs Non-Linear Shrinkage and all the variants of DCC-NLS (2024)
6. Large dynamic covariance matrices (2019)
7. Log-gases and random matrices (LMS-34) (2010)
8. Quadratic shrinkage for large covariance matrices (2022)

4 of 112 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.