Reinforcement Learning in Queue-Reactive Models: Application to Optimal Execution

Paper: arXiv 2511.15262

Authors: Tomas Espana, Yadh Hafsi, Fabrizio Lillo, Edoardo Vittori

Abstract

We investigate the use of Reinforcement Learning for the optimal execution of meta-orders, where the objective is to execute incrementally large orders while minimizing implementation shortfall and market impact over an extended period of time. Departing from traditional parametric approaches to price dynamics and impact modeling, we adopt a model-free, data-driven framework. Since policy optimization requires counterfactual feedback that historical data cannot provide, we employ the Queue-Reactive Model to generate realistic and tractable limit order book simulations that encompass transient price impact, and nonlinear and dynamic order flow responses. Methodologically, we train a Double Deep Q-Network agent on a state space comprising time, inventory, price, and depth variables, and evaluate its performance against established benchmarks. Numerical simulation results show that the agent learns a policy that is both strategic and tactical, adapting effectively to order book conditions and outperforming standard approaches across multiple training configurations. These findings provide strong evidence that model-free Reinforcement Learning can yield adaptive and robust solutions to the optimal execution problem.

Complexity vs Empirical Score

• Math Complexity: 8.5/10
• Empirical Rigor: 6.0/10 (capped from 8.0: no out-of-sample test; see penalty flags)
• Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper employs advanced mathematical concepts like stochastic control theory, HJB equations, viscosity solutions, and transient impact models with arbitrage constraints, indicating high mathematical complexity. It also demonstrates strong empirical rigor through the use of a calibrated Queue-Reactive Model simulator for training and evaluation, a standard RL architecture (DDQN), and comparative benchmarking in simulated environments, making it backtest-ready.

Research Flowchart

  flowchart TD
  Start["Research Goal:<br>Optimal Execution via RL"] --> Inputs["Key Inputs:<br>Limit Order Book Data"]
  
  Inputs --> Model["Methodology:<br>Queue-Reactive Model Simulation"]
  
  Model --> StateSpace["State Space Construction:<br>Time, Inventory, Price, Depth"]
  
  StateSpace --> DQN["Computational Process:<br>Train Double Deep Q-Network"]
  
  DQN --> Findings["Key Findings:<br>Adaptive Policy & Outperforms Benchmarks"]

Cited References (151)

1. Optimal execution in a time varying environment: well posedness and price manipulation (2024)
2. Markov decision processes: discrete stochastic dynamic programming (2014)
3. Human-level control through deep reinforcement learning (2015)
4. Dynamic programming (1966)
5. Optimal signal-adaptive trading with temporary and transient price impact (2020) — link
6. Optimal trading with stochastic liquidity and volatility (2012)
7. Optimal execution strategies in limit order books with general shape functions (2010)
8. Optimal trade execution and absence of price manipulations in limit order book models (2010)

10 of 151 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.