Paper: arXiv 2507.06345

Authors: Patrick Cheridito, Moritz Weiss

Abstract

In this paper, we introduce a novel reinforcement learning framework for optimal trade execution in a limit order book. We formulate the trade execution problem as a dynamic allocation task whose objective is the optimal placement of market and limit orders to maximize expected revenue. By employing multivariate logistic-normal distributions to model random allocations, the framework enables efficient training of the reinforcement learning algorithm. Numerical experiments show that the proposed method outperforms traditional benchmark strategies in simulated limit order book environments featuring noise traders submitting random orders, tactical traders responding to order book imbalances, and a strategic trader seeking to acquire or liquidate an asset position.

Complexity vs Empirical Score

  • Math Complexity: 8.5/10
  • Empirical Rigor: 6.0/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper employs advanced mathematics including multivariate logistic-normal distributions, stochastic calculus for limit order book modeling, and policy gradient derivations for RL, scoring high on math complexity. It features substantial empirical rigor through simulated market environments with multiple agent types (noise, tactical, strategic) and comparison to benchmark strategies, though it lacks real-world backtests or datasets.

Research Flowchart

  flowchart TD
  A["Research Goal: Optimal Trade Execution<br>Maximize revenue, minimize market impact"] --> B["Data/Inputs:<br>Simulated Limit Order Book (LOB)"]
  B --> C["Methodology: RL Framework<br>with Logistic-Normal Distributions"]
  C --> D["Action:<br>Market & Limit Order Placement"]
  D --> E["Process: Dynamic Allocation<br>via Reinforcement Learning"]
  E --> F["Findings: Outperformance vs. Benchmarks<br>(Noise/Tactical/Strategic Traders)"]

Cited References (66)

  1. Adam: A method for stochastic optimization (2017) — link
  2. RLTE (2025) — link
  3. Policy gradient methods (2011)
  4. Optimal execution with nonlinear impact functions and trading-enhanced risk (2003)
  5. Optimal trading with stochastic liquidity and volatility (2012)
  6. Optimal trade execution under G eometric B rownian M otion in the A lmgren and C hriss framework (2011)
  7. Dynamical models of market impact and algorithms for order execution (2013)
  8. Optimal trading strategy and supply/demand dynamics (2013)
and 58 more
  1. Optimal control of execution costs (1998)
  2. Optimal trade execution under stochastic volatility and liquidity (2014)
  3. Optimal execution with non-linear transient market impact (2017)
  4. Optimal order execution under price impact: a hybrid model (2024)
  5. A stochastic model for order book dynamics (2010)
  6. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms (2022)
  7. Algorithmic trading in a microstructural limit order book model (2022)
  8. A mathematical approach to order book modeling (2013)
  9. Reinforcement learning for optimized trade execution (2006)
  10. Double deep Q -learning for optimal execution (2021)
  11. A reinforcement learning extension to the A lmgren– C hriss framework for optimal trade execution (2014)
  12. Deep reinforcement learning for the optimal placement of cryptocurrency limit orders (2022)
  13. A reinforcement learning approach to optimal execution (2022)
  14. A mathematical framework for modelling order book dynamics — → 2302.01169
  15. ABIDES: Towards high-fidelity multi-agent market simulation (2020)
  16. Limit order books (2013)
  17. Modelling intensities of order flows in a limit order book (2017)
  18. Simulating and analyzing order book data: The queue-reactive model (2015)
  19. A single queue reactive Hawkes model for the order flow (2023)
  20. Optimal execution with limit and market orders (2015)
  21. Optimal execution of portfolio transactions (2001)
  22. Optimal portfolios from ordering information (2007)
  23. Deep reinforcement learning for market making under a hawkes process-based limit order book model (2022)
  24. Reinforcement L earning: An I ntroduction (2018)
  25. Incorporating order-flow into optimal execution (2016)
  26. General intensity shapes in optimal liquidation (2015)
  27. Algorithmic and H igh- F requency T rading (2015)
  28. Trades, Q uotes and P rices: F inancial M arkets U nder the M icroscope (2018)
  29. Monte carlo gradient estimation in machine learning (2020)
  30. Logistic-normal distributions: Some properties and uses (1980)
  31. High-dimensional continuous control using generalized advantage estimation (2015) — link
  32. Excluding the Irrelevant: Focusing Reinforcement Learning through Continuous Action Masking (2024) — link
  33. What matters for on-policy deep actor-critic methods? a large-scale study (2021)
  34. Simplex Decomposition for Portfolio Allocation Constraints in Reinforcement Learning. (2023)
  35. A prescriptive D irichlet power allocation policy with deep reinforcement learning (2022)
  36. Advances in Compositional Data Analysis (2021)
  37. The normal distribution in some constrained sample spaces (2008) — link
  38. The concrete distribution: A continuous relaxation of discrete random variables (2016) — link
  39. Policy gradient methods for reinforcement learning with function approximation (1999)
  40. Optimal order placement in limit order markets (2017)
  41. Learning a functional control for high-frequency finance (2022)
  42. Multi-agent reinforcement learning in a realistic limit order book market simulation (2020)
  43. Asynchronous D eep D ouble D ueling Q -learning for trading-signal execution in limit order book markets (2023)
  44. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks (2013) — link
  45. Practical Application of Deep Reinforcement Learning to Optimal Trade Execution (2023)
  46. Optimal Execution with Reinforcement Learning — → 2411.06389
  47. Actor-Critic Algorithms (1999)
  48. The micro-price: a high-frequency estimator of future prices (2018)
  49. Benchmarking deep reinforcement learning for continuous control (2016)
  50. Trust region policy optimization (2015)
  51. Proximal policy optimization algorithms (2017) — link
  52. No-dynamic-arbitrage and market impact (2010)
  53. Optimal portfolio execution using market and limit orders (2014)
  54. Optimal portfolio liquidation with limit orders (2012)
  55. Algorithmic trading with learning (2016)
  56. Optimal high-frequency trading in a pro rata microstructure with predictive information (2015)
  57. Optimal high-frequency trading with limit and market orders (2013)
  58. Modelling asset prices for algorithmic and high-frequency trading (2013)

2 of 66 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.