Paper: arXiv 2511.17304

Authors: Jian’an Zhang

Abstract

We study reinforcement learning (RL) on volatility surfaces through the lens of Scientific AI. We ask whether axiomatic no-arbitrage laws, imposed as soft penalties on a learned world model, can reliably align high-capacity RL agents, or mainly create Goodhart-style incentives to exploit model errors. From classical static no-arbitrage conditions we build a finite-dimensional convex volatility law manifold of admissible total-variance surfaces, together with a metric law-penalty functional and a Graceful Failure Index (GFI) that normalizes law degradation under shocks. A synthetic generator produces law-consistent trajectories, while a recurrent neural world model trained without law regularization exhibits structured off-manifold errors. On this testbed we define a Goodhart decomposition (r = r^{"\mathcal{M"}} + r^\perp), where (r^\perp) is ghost arbitrage from off-manifold prediction error. We prove a ghost-arbitrage incentive theorem for PPO-type agents, a law-strength trade-off theorem showing that stronger penalties eventually worsen P&L, and a no-free-lunch theorem: under a law-consistent world model and law-aligned strategy class, unconstrained law-seeking RL cannot Pareto-dominate structural baselines on P&L, penalties, and GFI. In experiments on an SPX/VIX-like world model, simple structural strategies form the empirical law-strength frontier, while all law-seeking RL variants underperform and move into high-penalty, high-GFI regions. Volatility thus provides a concrete case where reward shaping with verifiable penalties is insufficient for robust law alignment.

Complexity vs Empirical Score

  • Math Complexity: 9.5/10
  • Empirical Rigor: 4.0/10
  • Quadrant: Lab Rats — theoretically deep, empirically untested

Why this score: The paper is dense with advanced theoretical results, including proofs of incentive and trade-off theorems, convex manifolds, and Goodhart decomposition, indicating very high mathematical complexity. Empirically, while it uses a synthetic testbed and specific metrics (e.g., GFI), the work is primarily theoretical and diagnostic with no real-world data, backtests, or code, placing it lower on empirical rigor.

Research Flowchart

  flowchart TD
  A["Research Goal"] --> B["Data: Law-Consistent Trajectories"]
  B --> C["Methodology: RL on World Model"]
  C --> D["Computations"]
  subgraph D [" "]
      D1["Train RNN World Model"]
      D2["Apply Law Penalty"]
      D3["Run PPO Agent"]
  end
  D --> E{"Outcomes"}
  E --> F["Ghost Arbitrage Incentive"]
  E --> G["Law-Strength Trade-off"]
  E --> H["No Free Lunch on Volatility Frontier"]

Cited References (80)

  1. Pricing with a Smile (1994)
  2. The Volatility Surface: A Practitioner’s Guide (2006)
  3. Semiparametric Modeling of Implied Volatility (2005)
  4. Option Valuation Using the Fast Fourier Transform (1999)
  5. The Moment Formula for Implied Volatility at Extreme Strikes (2004)
  6. Financial Modelling with Jump Processes (2004)
  7. Stochastic Volatility Modeling (2016)
  8. Arbitrage-Free SVI Volatility Surfaces (2014)
and 72 more
  1. Pricing under Rough Volatility (2016)
  2. Deep Learning Volatility: Deep Calibration of Rough Stochastic Volatility Models (2021)
  3. Deep Learning Volatility (2021)
  4. Calibration of Local Stochastic Volatility Models to Market Data (2015)
  5. Linking Vanillas and VIX Options: A Constrained Optimization Approach (2015)
  6. A Sparse Modeling Approach to the Arbitrage-Free Interpolation of Plain-Vanilla Option Prices and Implied Volatilities (2023)
  7. Neural Network-Based Option Pricing (2020) — link
  8. A Note on Sufficient Conditions for No Arbitrage (2005)
  9. Interpolation Methods for Curve Construction (2006)
  10. Term-Structure Models: A Graduate Course (2009)
  11. Consistency Problems for Heath–Jarrow–Morton Interest Rate Models (2001)
  12. Interest Rate Dynamics and Consistent Forward Rate Curves (1999)
  13. Coherent Measures of Risk (1999)
  14. Optimization of Conditional Value-at-Risk (2000)
  15. Algorithmic and High-Frequency Trading (2015)
  16. A Century of Evidence on Trend-Following Investing (2017)
  17. Deep Reinforcement Learning for Trading (2020)
  18. Convex Analysis and Monotone Operator Theory in Hilbert Spaces (2011)
  19. Stochastic Calculus and Applications (2015)
  20. Manifold Regularization: A Geometric Framework for Learning from Labeled and Unlabeled Examples (2006)
  21. Testing the Manifold Hypothesis (2016)
  22. Solving High-Dimensional Partial Differential Equations Using Deep Learning (2018)
  23. Machine Learning Approximation Algorithms for High-Dimensional Fully Nonlinear Partial Differential Equations and Second-Order Backward SDE s (2019)
  24. Physics-Informed Neural Networks: A Deep Learning Framework for Solving Forward and Inverse Problems Involving Nonlinear Partial Differential Equations (2019)
  25. Physics-Informed Neural Networks: A Deep Learning Framework for Solving Forward and Inverse Problems Involving Nonlinear Partial Differential Equations (2019)
  26. Physics-Informed Machine Learning (2021)
  27. Machine Learning and the Physical Sciences (2019)
  28. Integrating Physics-Based Modeling with Machine Learning: A Survey (2020)
  29. Solving Stochastic Differential Equations and Kolmogorov Equations by Deep Learning (2021)
  30. Message Passing Neural PDE Solvers (2022)
  31. Relational Inductive Biases, Deep Learning, and Graph Networks (2018) — link
  32. World Models (2018) — link
  33. Learning Latent Dynamics for Planning from Pixels (2019)
  34. Dream to Control: Learning Behaviors by Latent Imagination (2020)
  35. Mastering Diverse Domains with World Models (2023) — link
  36. Deep Reinforcement Learning in a Handful of Trials Using Probabilistic Dynamics Models (2018)
  37. When to Trust Your Model: Model-Based Policy Optimization (2019)
  38. Model-Based Reinforcement Learning for Atari (2020)
  39. Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model (2020)
  40. Human-Level Control Through Deep Reinforcement Learning (2015)
  41. Reinforcement Learning: An Introduction (2018)
  42. Actor-Critic Algorithms (2000)
  43. Bias in Natural Actor-Critic Algorithms (2014)
  44. High-Dimensional Continuous Control Using Generalized Advantage Estimation (2016)
  45. Proximal Policy Optimization Algorithms (2017)
  46. Constrained Markov Decision Processes (1999)
  47. Constrained Policy Optimization (2017)
  48. Mean-Variance Optimization in Markov Decision Processes (2011)
  49. Risk-Sensitive and Robust Decision-Making: A CVaR Optimization Approach (2015)
  50. A Survey of Multi-Objective Sequential Decision-Making (2013)
  51. A Unified View of Entropy-Regularized Markov Decision Processes (2017)
  52. A Theory of Regularized Markov Decision Processes (2019)
  53. Towards Safe Reinforcement Learning: A Survey and Outlook (2021) — link
  54. Regularized Policy Optimization in MDP s with Constraints (2021)
  55. Learning to Trade via Direct Reinforcement (2001)
  56. Deep Direct Reinforcement Learning for Financial Signal Representation and Trading (2016)
  57. A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem (2017) — link
  58. FinRL : A Deep Reinforcement Learning Library for Quantitative Finance (2021)
  59. Dynamic Replication and Hedging: A Reinforcement Learning Approach (2019)
  60. Deep Hedging (2019)
  61. Deep Hedging (2019)
  62. Deep Hedging (2019)
  63. Deep Hedging of Derivatives with Transaction Costs and Different Risk Criteria (2020)
  64. DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks (2020)
  65. A Transformer-Based Framework for Multivariate Time Series Representation Learning (2021)
  66. Categorizing Variants of Goodhart’s Law (2019) — link
  67. Concrete Problems in AI Safety (2016) — link
  68. AI Safety Gridworlds (2017)
  69. Reinforcement Learning with a Corrupted Reward Channel (2017) — link
  70. Hidden Incentives in Deep Reinforcement Learning (2020)
  71. The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models (2022) — link
  72. A Practical Guide to Multi-Objective Reinforcement Learning and Planning (2022)

0 of 80 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.