Paper: arXiv 2408.11773

Abstract

The use of reinforcement learning algorithms in financial trading is becoming increasingly prevalent. However, the autonomous nature of these algorithms can lead to unexpected outcomes that deviate from traditional game-theoretical predictions and may even destabilize markets. In this study, we examine a scenario in which two autonomous agents, modeled with Double Deep Q-Learning, learn to liquidate the same asset optimally in the presence of market impact, using the Almgren-Chriss (2000) framework. Our results show that the strategies learned by the agents deviate significantly from the Nash equilibrium of the corresponding market impact game. Notably, the learned strategies exhibit tacit collusion, closely aligning with the Pareto-optimal solution. We further explore how different levels of market volatility influence the agents’ performance and the equilibria they discover, including scenarios where volatility differs between the training and testing phases.

Complexity vs Empirical Score

  • Math Complexity: 7.5/10
  • Empirical Rigor: 6.0/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper employs advanced mathematical frameworks like the Almgren-Chriss market impact model, game-theoretic Nash and Pareto-optimal equilibria derivations, and reinforcement learning theory, indicating high mathematical complexity. Empirically, it uses simulated trading environments with Double Deep Q-Learning, compares learned strategies to theoretical benchmarks, and tests robustness under varying volatility regimes, demonstrating substantial data and implementation rigor.

Research Flowchart

  flowchart TD
  A["Research Goal: <br>How do RL agents in optimal execution <br>deviate from Nash Equilibrium?"] --> B["Methodology: <br>Double Deep Q-Learning Agents"]
  B --> C["Data/Inputs: <br>Almgren-Chriss Framework <br>with Market Impact & Volatility"]
  C --> D["Computational Process: <br>Training two autonomous agents <br>to liquidate same asset"]
  D --> E["Outcome 1: <br>Significant deviation <br>from Nash Equilibrium"]
  D --> F["Outcome 2: <br>Emergence of Tacit Collusion <br>(Pareto-optimal strategy)"]
  D --> G["Outcome 3: <br>Volatility-dependent performance <br>and equilibria discovery"]

Cited References (95)

  1. Optimal execution in a time varying environment: well posedness and price manipulation (2024)
  2. Optimal signal-adaptive trading with temporary and transient price impact (2020) — link
  3. Optimal trading with stochastic liquidity and volatility (2012)
  4. Optimal execution strategies in limit order books with general shape functions (2010)
  5. Optimal trade execution and absence of price manipulations in limit order book models (2010)
  6. Optimal execution: A review (2022)
  7. Robust strategies for optimal order execution in the Almgren–Chriss framework (2013)
  8. Optimal execution with non-linear transient market impact (2017)
and 87 more
  1. Transient linear price impact and Fredholm integral equations (2012)
  2. Optimal portfolio liquidation with limit orders (2012)
  3. Liquidation in limit order books with controlled intensity (2014)
  4. Optimal execution of portfolio transactions (2000)
  5. Optimal control of execution costs (1998)
  6. Incorporating signals into optimal trading (2019)
  7. General intensity shapes in optimal liquidation (2015)
  8. Incorporating order-flow into optimal execution (2016)
  9. Trading algorithms with learning in latent alpha models (2019)
  10. Reinforcement learning for optimized trade execution (2006)
  11. A reinforcement learning extension to the Almgren-Chriss framework for optimal trade execution (2014)
  12. Q-learning (1992)
  13. Playing atari with deep reinforcement learning (2013) — link
  14. Deep reinforcement learning with double q-learning (2016)
  15. Double deep q-learning for optimal execution (2021)
  16. Improving financial trading decisions using deep Q-learning: Predicting the number of shares, action strategies, and transfer learning (2019)
  17. Deep execution-value and policy based reinforcement learning for trading and beating market benchmarks (2019)
  18. Reinforcement learning for quantitative trading (2023)
  19. Recent advances in reinforcement learning in finance (2023)
  20. No-dynamic-arbitrage and market impact (2010)
  21. Price manipulation and quasi-arbitrage (2004)
  22. Adaptive market making via online learning (2013)
  23. Deep reinforcement learning for market making (2020)
  24. Model-based deep reinforcement learning for dynamic portfolio optimization (2019) — link
  25. Machine learning for market microstructure and high frequency trading (2013)
  26. Model-based reinforcement learning for predictions and control for limit order books (2019) — link
  27. Deep reinforcement learning for active high frequency trading (2021) — link
  28. Multi-agent reinforcement learning in a realistic limit order book market simulation (2020)
  29. Deep reinforcement learning for the optimal placement of cryptocurrency limit orders (2022)
  30. Continuous auctions and insider trading (1985)
  31. Deep differentiable reinforcement learning and optimal trading (2022)
  32. Optimal trading strategy and supply/demand dynamics (2013)
  33. Optimal trading with signals and stochastic price impact (2022)
  34. Liquidity fluctuations and the latent dynamics of price impact (2022)
  35. Optimal control of execution costs (1998)
  36. Direct estimation of equity market impact (2005)
  37. Fluctuations and response in financial markets: the subtle nature ofrandom’price changes (2003)
  38. How markets slowly digest changes in supply and demand (2009)
  39. Optimal basket liquidation for CARA investors is deterministic (2010)
  40. Algorithmic and high-frequency trading (2015)
  41. Reinforcement learning: an introduction MIT Press (1998)
  42. Optimal liquidation under stochastic price impact (2019)
  43. Measuring price impact and information content of trades in a time-varying setting — → 2212.12687
  44. Reinforcement Learning for Optimal Execution when Liquidity is Time-Varying — → 2402.12049
  45. A State-Constrained Differential Game Arising in Optimal Portfolio Liquidation (2017)
  46. Dynamics of market making algorithms in dealer markets: Learning and tacit collusion (2022)
  47. Interbank lending with benchmark rates: Pareto optima for a class of singular control games (2021)
  48. Episodic liquidity crises: Cooperative and predatory trading (2007)
  49. Predatory trading (2005)
  50. Predatory trading: a game on volatility and liquidity (2011)
  51. Liquidation in the face of adversity: stealth vs. sunshine trading (2009)
  52. A market impact game under transient price impact (2019)
  53. Closed-loop Nash competition for liquidity (2023)
  54. An FBSDE approach to market impact games with stochastic parameters (2019) — link
  55. Trading with the crowd (2023)
  56. Transient impact from the Nash equilibrium of a permanent market impact game (2024)
  57. Nonlinear multiobjective optimization (1999)
  58. Multi-agent deep reinforcement learning for liquidation strategy analysis (2019) — link
  59. On the analytical derivation of the Pareto-optimal set with applications to structural design (2015)
  60. pymoo: Multi-Objective Optimization in Python (2020)
  61. Q-learning agents in a Cournot oligopoly model (2008)
  62. Artificial intelligence: Can seemingly collusive outcomes be avoided? (2023)
  63. Algorithmic collusion: Insights from deep learning (2021)
  64. Interactions of market making algorithms (2022)
  65. Algorithmic collusion in electronic markets: The impact of tick size (2022)
  66. Algorithmic trading methods: Applications using advanced statistics, optimization, and machine learning techniques (2020)
  67. Co-impact: Crowding effects in institutional trading activity (2020)
  68. Instabilities in multi-asset and multi-agent market impact games (2024)
  69. Artificial intelligence, algorithmic pricing, and collusion (2020)
  70. Algorithmic collusion: A critical review (2021) — link
  71. Collusion by mistake: Does algorithmic sophistication drive supra-competitive profits? (2024)
  72. The theory of industrial organization (1988)
  73. Ai-powered trading, algorithmic collusion, and price efficiency (2024)
  74. Algorithmic pricing and liquidity in securities markets (2022)
  75. Human-level control through deep reinforcement learning (2015)
  76. Game theory (1991)
  77. Deep Q-learning for Nash equilibria: Nash-DQN (2022)
  78. Multi-Agent Reinforcement Learning for Greenhouse Gas Offset Credit Markets — → 2504.11258
  79. Machine Learning, Market Manipulation, and Collusion on Capital Markets: Why the" Black Box" Matters (2021)
  80. Approximation by superpositions of a sigmoidal function (1989)
  81. Learning and pricing with models that do not explicitly incorporate competition (2015)
  82. Anonymity, Signaling, and Collusion in Limit Order Books (2025)
  83. Spoofing and Manipulating Order Books with Learning Algorithms (2023)
  84. Less than meets the eye: simultaneous experiments as a source of algorithmic seeming collusion (2024)
  85. Developing competition law for collusion by autonomous artificial agents (2018)
  86. Survey of self-play in reinforcement learning (2021) — link
  87. High-frequency limit of Nash equilibria in a market impact game with transient price impact (2017)

3 of 95 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.