Paper: arXiv 2510.22206

Authors: Ollie Olby, Andreea Bacalum, Rory Baggott, Namid Stillman

Abstract

Execution algorithms are vital to modern trading, they enable market participants to execute large orders while minimising market impact and transaction costs. As these algorithms grow more sophisticated, optimising them becomes increasingly challenging. In this work, we present a reinforcement learning (RL) framework for discovering optimal execution strategies, evaluated within a reactive agent-based market simulator. This simulator creates reactive order flow and allows us to decompose slippage into its constituent components: market impact and execution risk. We assess the RL agent’s performance using the efficient frontier based on work by Almgren and Chriss, measuring its ability to balance risk and cost. Results show that the RL-derived strategies consistently outperform baselines and operate near the efficient frontier, demonstrating a strong ability to optimise for risk and impact. These findings highlight the potential of reinforcement learning as a powerful tool in the trader’s toolkit.

Complexity vs Empirical Score

  • Math Complexity: 5.5/10
  • Empirical Rigor: 7.0/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper presents a reinforcement learning framework with defined mathematical formulations like the Almgren-Chriss efficient frontier and slippage decomposition, indicating moderate math complexity. It employs an agent-based market simulator for backtesting, a concrete implementation detail that demonstrates high empirical rigor.

Research Flowchart

  flowchart TD
  A["Research Goal: Optimise<br>Execution Algorithms"] --> B["Methodology: Reinforcement Learning<br>in Agent-Based Market Simulator"]
  B --> C["Inputs: Reactive Order Flow<br>Order Book Data"]
  C --> D["Process: RL Agent Learning<br>Slippage Decomposition"]
  D --> E["Evaluation: Efficient Frontier<br>Almgren-Chriss Metrics"]
  E --> F["Outcome: RL Strategies Outperform<br>Baselines Near Efficient Frontier"]

Cited References (32)

  1. A reinforcement learning extension to the Almgren-Chriss framework for optimal trade execution (2014)
  2. Hedging using reinforcement learning: Contextual k-armed bandit versus Q-learning (2023)
  3. Survey on applications of multi-armed and contextual bandits (2020)
  4. Optimal execution of portfolio transactions (2000)
  5. Reinforcement Learning for Optimized Trade Execution (2006)
  6. Reinforcement Learning in Agent-Based Market Simulation: Unveiling Realistic Stylized Facts and Behavior (2024) — link
  7. Playing Atari with Deep Reinforcement Learning (2013) — link
  8. OpenAI Gym (2016) — link
and 24 more
  1. An end-to-end optimal trade execution framework based on proximal policy optimization (2021)
  2. Optimal Execution with Reinforcement Learning (2024) — link
  3. Deep Reinforcement Learning in Non-Markov Market-Making (2025)
  4. VWAP execution as an optimal strategy (2015)
  5. A Modular Framework for Reinforcement Learning Optimal Execution (2022) — link
  6. Static vs adapted optimal execution strategies in two benchmark trading models (2016) — link
  7. Intraday periodicity in algorithmic trading (2014) — link
  8. The intraday patterns of the spread and depth in a market without market makers: The Stock Exchange of Hong Kong (1999)
  9. Simple statistical gradient-following algorithms for connectionist reinforcement learning (1992)
  10. Co-existence of Trend and Value in Financial Markets: Estimating an Extended Chiarella Model (2018) — link
  11. A simulation analysis of the microstructure of double auction markets (2002) — link
  12. The predictive power of zero intelligence in financial markets (2005) — link
  13. Allocative Efficiency of Markets with Zero-Intelligence Traders: Market as a Partial Substitute for Individual Rationality (1993) — link
  14. Multi-agent–based Order Book Model of financial markets (2006) — link
  15. ABIDES: Towards High-Fidelity Market Simulation for AI Research (2019) — link
  16. Learning to simulate realistic limit order book markets from data as a World Agent (2022) — link
  17. Get Real: Realism Metrics for Robust Limit Order Book Market Simulations (2019) — link
  18. Fast Agent-Based Simulation Framework with Applications to Reinforcement Learning and the Study of Trading Latency Effects (2022) — link
  19. How to Evaluate Trading Strategies: Single Agent Market Replay or Multiple Agent Interactive Simulation? (2019) — link
  20. Agent-based Liquidity Risk Modelling for Financial Markets (2025) — link
  21. Deep Calibration of Market Simulations using Neural Density Estimators and Embedding Networks (2023) — link
  22. Trades, Quotes and Prices: Financial Markets Under the Microscope (2018)
  23. Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act) (2024) — link
  24. A Comprehensive Survey of Reinforcement Learning: From Algorithms to Practical Challenges (2025) — link

0 of 32 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.