Paper: arXiv 2411.18830

Abstract

We study the relationship between model complexity and out-of-sample performance in the context of mean-variance portfolio optimization. Representing model complexity by the number of assets, we find that the performance of low-dimensional models initially improves with complexity but then declines due to overfitting. As model complexity becomes sufficiently high, the performance improves with complexity again, resulting in a double ascent Sharpe ratio curve similar to the double descent phenomenon observed in artificial intelligence. The underlying mechanisms involve an intricate interaction between the theoretical Sharpe ratio and estimation accuracy. In high-dimensional models, the theoretical Sharpe ratio approaches its upper limit, and the overfitting problem is reduced because there are more parameters than data restrictions, which allows us to choose well-behaved parameters based on inductive bias.

Complexity vs Empirical Score

  • Math Complexity: 8.5/10
  • Empirical Rigor: 3.0/10
  • Quadrant: Lab Rats — theoretically deep, empirically untested

Why this score: The paper employs advanced theoretical frameworks like random matrix theory and derives analytical asymptotic results for Sharpe ratios, indicating high mathematical complexity. However, the excerpt focuses on theoretical derivations and conceptual mechanisms without showing specific backtesting procedures, code, or detailed empirical metrics, resulting in lower empirical rigor.

Research Flowchart

  flowchart TD
  A["Research Goal: Investigate relationship between model complexity (number of assets) and out-of-sample Sharpe ratio in mean-variance portfolio optimization"] --> B["Data: Historical equity returns data (N assets, T time periods)"]
  
  B --> C["Methodology: Simulate varying model complexity (N from low to high)"]
  
  C --> D["Computational Process: Construct Mean-Variance portfolios using empirical inputs (sample mean/covariance) across all complexity levels"]
  
  D --> E["Analysis: Evaluate out-of-sample Sharpe ratio performance against model complexity"]
  
  E --> F["Finding 1: Performance improves initially with complexity (low N)"]
  E --> G["Finding 2: Performance declines due to overfitting (moderate N)"]
  E --> H["Finding 3: Performance improves again with high complexity (N > T)"]
  
  F --> I["Key Outcome: Double Descent Sharpe Ratio Curve<br/>Mechanism: Interaction between theoretical Sharpe limit & estimation accuracy in high dimensions"]
  G --> I
  H --> I

Cited References (120)

  1. Approaching mean-variance efficiency for large portfolios (2019)
  2. Market efficiency in the age of big data (2022)
  3. Lest we forget: Learn from out-of-sample forecast errors when optimizing portfolios (2022)
  4. A transaction-cost perspective on the multitude of firm characteristics (2020)
  5. Empirical cross-sectional asset pricing: a survey (2012)
  6. Presidential address: Discount rates (2011)
  7. The statistical limit of arbitrage (2024)
  8. Dissecting characteristics nonparametrically (2020)
and 112 more
  1. Parametric portfolio policies: Exploiting characteristics in the cross-section of equity returns (2009)
  2. On asymptotics of eigenvectors of large sample covariance matrix (2007) — link
  3. Asymptotic properties of eigenmatrices of a large sample covariance matrix (2011) — link
  4. Local operator theory, random matrices and Banach spaces (2001)
  5. Matrix analysis (2012)
  6. Statistics on special manifolds (2003)
  7. Multivariate analysis (1979)
  8. Topics in matrix analysis (1994)
  9. The distribution of eigenvalues in certain sets of random matrices (1967)
  10. The strong limits of random matrix spectra for sample matrices of independent elements (1978)
  11. Strong convergence of the empirical distribution of eigenvalues of large dimensional random matrices (1995)
  12. High-dimensionality effects in the Markowitz problem and other quadratic programs with linear constraints: Risk underestimation (2010) — link
  13. Random matrix theory and robust covariance matrix estimation for financial data (2005)
  14. Risk and asset allocation (2005)
  15. Quantitative risk management: concepts, techniques and tools-revised edition (2015)
  16. The Student ensemble of correlation matrices: eigenvalue spectrum and Kullback-Leibler entropy (2007) — link
  17. Spectral analysis of large dimensional random matrices (2010)
  18. On the empirical distribution of eigenvalues of a class of large dimensional random matrices (1995)
  19. Spectral convergence for a general class of random matrices (2011)
  20. Overparametrized Regression: Ridgeless Interpolation (2023)
  21. Surprises in high-dimensional ridgeless least squares interpolation (2022)
  22. On asymptotics of eigenvectors of large sample covariance matrix (2007)
  23. Central limit theorem for hotelling’s T 2 statistic under large dimension (2011)
  24. Portfolio Selection (1952) — link
  25. Distribution of eigenvalues for some sets of random matrices (1967)
  26. An introduction to multivariate statistical analysis (1958)
  27. Estimation for Markowitz efficient portfolios (1980)
  28. Economic Forecasts Using Many Noises (2023) — link
  29. Asset allocation under distribution uncertainty (2004)
  30. Portfolio theory: as I still see it (2010)
  31. A brief review of portfolio optimization techniques (2023)
  32. Robust portfolio optimization: a categorized bibliographic review (2020)
  33. Optimal portfolio choice with unknown benchmark efficiency (2024)
  34. Forest through the trees: Building cross-sections of stock returns (2023)
  35. Corporate investment and asset price dynamics: Implications for the cross-section of returns (2004)
  36. The value premium (2005)
  37. A generalized approach to portfolio optimization: Improving performance by constraining portfolio norms (2009)
  38. Estimation Risk and OptimalPortfolio Choice North-Holland (1979)
  39. Risk reduction in large portfolios: Why imposing the wrong constraints helps (2003)
  40. Keynes Meets Markowitz: The Trade of Between Familiarity and Diversification (2010)
  41. Markowitz meets Talmud: A combination of sophisticated and naive diversification strategies (2011)
  42. Anomalies and the expected market return (2022)
  43. Universal Portfolio Shrinkage (2024)
  44. The virtue of complexity everywhere (2022)
  45. APT or “ AIPT ”? The Surprising Dominance of Large Factor Models (2024)
  46. Structural deep learning in conditional asset pricing (2023)
  47. The virtue of complexity in return prediction (2024)
  48. Machine learning and the implementable efficient frontier (2024)
  49. Factor models, machine learning, and asset pricing (2022)
  50. Financial machine learning (2023)
  51. Machine learning vs. economic restrictions: Evidence from stock return predictability (2023)
  52. AlphaPortfolio: Direct construction through deep reinforcement learning and interpretable AI (2022)
  53. Asset pricing with panel tree under global split criteria (2023)
  54. Portfolio choices with many big models (2022)
  55. Nonlinear shrinkage of the covariance matrix for portfolio selection: Markowitz meets Goldilocks (2017)
  56. Portfolio selection with parameter and model uncertainty: A multi-prior approach (2007)
  57. Optimal versus naive diversification: How inefficient is the 1/ N portfolio strategy? (2009)
  58. The effect of errors in means, variances, and covariances on optimal portfolio choice (1993)
  59. Complexity in factor pricing models (2023)
  60. Time-varying minimum variance portfolio (2022)
  61. Factor models for portfolio selection in large dimensions: The good, the better and the ugly (2021)
  62. Sparse and stable Markowitz portfolios (2009)
  63. Benign overfitting in linear regression (2020)
  64. Eigenvectors of some large sample covariance matrix ensembles (2011)
  65. A regularized Hotelling’s T 2 test for pathway analysis in proteomic studies (2011)
  66. Empirical asset pricing via machine learning (2020)
  67. Anisotropic local laws for random matrices (2017)
  68. Generalized inversion of modified matrices (1973)
  69. Large sample covariance matrices without independence structures in columns (2008)
  70. Limiting spectral distribution for large sample covariance matrices with m-dependent elements (2010)
  71. The limiting spectral distribution for large sample covariance matrices with unbounded m-dependent entries (2016)
  72. Gaussian fluctuations for sample covariance matrices with dependent data (2013)
  73. Reconciling modern machine-learning practice and the classical bias–variance trade-off (2019)
  74. Harmless interpolation of noisy data in regression (2020)
  75. Two models of double descent for weak features (2020)
  76. Understanding overfitting peaks in generalization error: Analytical risk curves for $ l_2 $ and $ l_1 $ penalized interpolation (2019) — link
  77. JUST INTERPOLATE: KERNEL “RIDGELESS” REGRESSION CAN GENERALIZE (2020)
  78. The generalization error of random features regression: Precise asymptotics and the double descent curve (2022)
  79. Deep learning: a statistical viewpoint (2021)
  80. Exact expressions for double descent and implicit regularization via surrogate random design (2020)
  81. On the role of optimization in double descent: A least squares study (2021)
  82. A farewell to the bias-variance tradeoff? an overview of the theory of overparameterized machine learning (2021) — link
  83. High-dimensional dynamics of generalization error in neural networks (2020)
  84. Linear and nonlinear extension of the pseudo-inverse solution for learning boolean functions (1989)
  85. Benign overfitting and noisy features (2023)
  86. Autoencoder asset pricing models (2021)
  87. Deep learning in asset pricing (2024)
  88. The sampling error in estimates of mean-variance efficient portfolio weights (1999)
  89. Mean–variance portfolio optimization with deep learning based-forecasts for cointegrated stocks (2022)
  90. Optimal portfolio choice with parameter uncertainty (2007)
  91. APT or “AIPT” ? The Surprising Dominance of Large Factor Models (2024)
  92. Growing the efficient frontier on panel trees (2022)
  93. Time-series and cross-sectional stock return forecasting: New machine learning methods (2020)
  94. Bond risk premiums with machine learning (2021)
  95. A well-conditioned estimator for large-dimensional covariance matrices (2004)
  96. Estimation of the global minimum variance portfolio in high dimensions (2018)
  97. Two is better than one: Regularized shrinkage of large minimum variance portfolios (2024)
  98. Optimal portfolio choice with estimation risk: No risk-free asset case (2022)
  99. The risk of expected utility under parameter uncertainty (2024)
  100. Machine learning in asset management—Part 2: Portfolio construction—Weight optimization (2020)
  101. Deep learning for portfolio optimization (2020) — link
  102. Portfolio optimization with 2D relative-attentional gated transformer (2020)
  103. Do Common Factors Really Explain the Cross-Section of Stock Returns? (2020)
  104. Efficient portfolio selection in a large market (2016)
  105. Eigen portfolio selection: A robust approach to sharpe ratio maximization (2018)
  106. Finite sample approximation results for principal component analysis: A matrix perturbation approach (2008)
  107. On the effect of noise on fitting linear regression models (2024) — link
  108. Deep double descent: Where bigger models and more data hurt (2021)
  109. Forecasting using principal components from a large number of predictors (2002)
  110. Inferential theory for factor models of large dimensions (2003)
  111. Large covariance estimation by thresholding principal orthogonal complements (2013)
  112. l2-Relaxation: With Applications to Forecast Combination and Portfolio Analysis (2022)

0 of 120 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.