Paper: arXiv 2505.19058

Authors: Chung I Lu, Julian Sester, Aijia Zhang

Abstract

We propose a novel distributionally robust $Q$-learning algorithm for the non-tabular case accounting for continuous state spaces where the state transition of the underlying Markov decision process is subject to model uncertainty. The uncertainty is taken into account by considering the worst-case transition from a ball around a reference probability measure. To determine the optimal policy under the worst-case state transition, we solve the associated non-linear Bellman equation by dualising and regularising the Bellman operator with the Sinkhorn distance, which is then parameterized with deep neural networks. This approach allows us to modify the Deep Q-Network algorithm to optimise for the worst case state transition. We illustrate the tractability and effectiveness of our approach through several applications, including a portfolio optimisation task based on S&{“P”}~500 data.

Complexity vs Empirical Score

  • Math Complexity: 9.0/10
  • Empirical Rigor: 7.5/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper employs advanced mathematics including dualization of the Bellman operator and Sinkhorn distances for robust optimization, and it demonstrates effectiveness through a portfolio optimization task on S&P 500 data, indicating substantial empirical rigor.

Research Flowchart

  flowchart TD
  A["Research Goal"] -->|Develop DRL algorithm robust to model uncertainty| B["Methodology: Distributionally Robust Q-Learning"]
  B --> C["Modeling Uncertainty"]
  C --> D["Solve Non-linear Bellman Equation"]
  D --> E["Optimization: Sinkhorn Distance + Dualization"]
  E --> F["Deep Q-Network Implementation"]
  F --> G["Experimental Application"]
  G --> H["Findings: Robustness & Performance"]
  H --> I["S&P 500 Portfolio Optimization"]

Cited References (84)

  1. Playing FPS games with deep reinforcement learning (2017)
  2. Asynchronous methods for deep reinforcement learning (2016)
  3. Deep reinforcement learning with double Q -learning (2016)
  4. Addressing function approximation error in actor-critic methods (2018)
  5. Optimally combining sampling techniques for Monte Carlo rendering (1995)
  6. Scrambling Sobol’and Niederreiter–Xing Points (1998)
  7. Continuous control with deep reinforcement learning (2016)
  8. Twice regularized Markov decision processes: The equivalence between robustness and regularization (2023) — link
and 76 more
  1. Robustness in Sequential Decision Making under Evolving Uncertainty: Evidence from High-Frequency Market Making (2026) — link
  2. Kullback-Leibler divergence constrained distributionally robust optimization (2013)
  3. Wasserstein distributionally robust optimization: Theory and applications in machine learning (2019)
  4. Optimal control of execution costs (1998) — link
  5. Human-level control through deep reinforcement learning (2015) — link
  6. Distributionally Robust Deep Q-Learning — → 2505.19058
  7. Rectangularity and duality of distributionally robust Markov decision processes (2023) — link
  8. Optimal transport: old and new (2009)
  9. Introduction to entropic optimal transport (2021)
  10. Principles of mathematical analysis (1964)
  11. Robust Q -learning algorithm for Markov decision processes under Wasserstein uncertainty (2024)
  12. Generative model for financial time series trained with MMD using a signature kernel — → 2407.19848
  13. Learning to utilize shaping rewards: A new approach of reward shaping (2020)
  14. Universal approximation results for neural networks with non-polynomial activation function over non-compact domains (2024) — link
  15. Dota 2 with large scale deep reinforcement learning (2019) — link
  16. Probability essentials (2012)
  17. Probability theory: a comprehensive course (2013)
  18. Double Q -learning (2010)
  19. On the theory of dynamic programming (1952)
  20. Q-learning (1992) — link
  21. Reinforcement learning: An introduction (2018)
  22. Computational aspects of robust optimized certainty equivalents and option pricing (2020)
  23. Distributionally robust Markov decision processes (2010)
  24. Sample complexity of variance-reduced distributionally robust Q -learning (2024)
  25. Infinite dimensional analysis: a hitchhiker’s guide (2006)
  26. Sinkhorn divergences for unbalanced optimal transport (2019) — link
  27. Nonparametric adaptive robust control under model uncertainty (2023)
  28. Robust Markov decision processes (2013)
  29. Improved sample complexity bounds for distributionally robust reinforcement learning (2023)
  30. Neural networks can detect model-free static arbitrage strategies — → 2306.16422
  31. Universal approximation with deep narrow networks (2020)
  32. Distributionally robust reinforcement learning (2019) — link
  33. A convex optimization approach to distributionally robust Markov decision processes with Wasserstein distance (2017)
  34. On linear optimization over Wasserstein balls (2022)
  35. Robust control of Markov decision processes with uncertain transition matrices (2005)
  36. Robust dynamic programming (2005)
  37. An Efficient Solution to s-Rectangular Robust Markov Decision Processes (2023) — link
  38. Non-concave distributionally robust stochastic control in a discrete time finite horizon setting — → 2404.05230
  39. Policy Gradient in Robust MDP s with Global Convergence Guarantee (2023) — link
  40. Interpolating between optimal transport and mmd using Sinkhorn divergences (2019)
  41. Convex optimization (2004)
  42. Policy Gradient Method For Robust Reinforcement Learning (2022) — link
  43. Markov decision processes under model uncertainty (2023)
  44. Distributionally robust Markov decision processes and their connection to risk measures (2022)
  45. Computational Optimal Transport: With Applications to Data Science (2019)
  46. Robust Markov Decision Processes: Beyond Rectangularity (2023) — link
  47. Policy gradient algorithms for robust MDP s with non-rectangular uncertainty sets (2023) — link
  48. Distributionally robust optimization: A review (2019) — link
  49. Distributionally robust convex optimization (2014)
  50. Robust MDPs with k-Rectangular Uncertainty (2016) — link
  51. A relationship between arbitrary positive matrices and doubly stochastic matrices (1964)
  52. Natural actor-critic for robust reinforcement learning with function approximation (2023)
  53. Online robust reinforcement learning with model uncertainty (2021)
  54. Mathematical foundations of distributionally robust multistage optimization (2021)
  55. Investigating the parameters of the beta distribution (2023)
  56. Sinkhorn distributionally robust optimization (2021) — link
  57. Maximum entropy RL (provably) solves some robust RL problems (2021) — link
  58. Distributionally Robust Q -Learning (2022)
  59. On the efficiency of entropic regularized algorithms for optimal transport (2022)
  60. Robust Q -Learning for finite ambiguity sets (2024) — link
  61. Robust reinforcement learning using offline data (2022)
  62. A theoretical analysis of deep Q -learning (2020)
  63. Distributionally robust model-based reinforcement learning with large state spaces (2024)
  64. Multi-stage distributionally robust stochastic dual dynamic programming to multi-period economic dispatch with virtual energy storage (2021)
  65. Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networks (1990)
  66. Sinkhorn Distances: Lightspeed Computation of Optimal Transport (2013) — link
  67. Universal approximation using feedforward neural networks: A survey of some existing methods, and some new results (1998)
  68. A finite sample complexity bound for distributionally robust Q -learning (2023)
  69. Wasserstein distributionally robust stochastic control: A data-driven approach (2020)
  70. Stability of entropic optimal transport and Schrödinger bridges (2022)
  71. Approximation theory of the MLP model in neural networks (1999)
  72. Deep reinforcement learning: From Q -learning to deep Q -learning (2017)
  73. Robust SGLD algorithm for solving non-convex distributionally robust optimisation problems — → 2403.09532
  74. Structural estimation of Markov decision processes (1994)
  75. Sample complexity of Sinkhorn divergences (2019)
  76. Empirical properties of asset returns: stylized facts and statistical issues (2001) — link

5 of 84 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.