Paper: arXiv 2402.12049

Abstract

Optimal execution is an important problem faced by any trader. Most solutions are based on the assumption of constant market impact, while liquidity is known to be dynamic. Moreover, models with time-varying liquidity typically assume that it is observable, despite the fact that, in reality, it is latent and hard to measure in real time. In this paper we show that the use of Double Deep Q-learning, a form of Reinforcement Learning based on neural networks, is able to learn optimal trading policies when liquidity is time-varying. Specifically, we consider an Almgren-Chriss framework with temporary and permanent impact parameters following several deterministic and stochastic dynamics. Using extensive numerical experiments, we show that the trained algorithm learns the optimal policy when the analytical solution is available, and overcomes benchmarks and approximated solutions when the solution is not available.

Complexity vs Empirical Score

  • Math Complexity: 7.5/10
  • Empirical Rigor: 4.0/10
  • Quadrant: Lab Rats — theoretically deep, empirically untested

Why this score: The paper employs advanced mathematical formulations, including stochastic differential equations and a Double Deep Q-learning neural network architecture, reflecting high mathematical density. However, the empirical validation relies on simulated data and numerical experiments rather than live trading, backtesting on real historical data, or public code repositories, resulting in lower empirical rigor.

Research Flowchart

  flowchart TD
  A["Research Goal<br>Optimal Execution with<br>Time-Varying Liquidity?"] --> B["Methodology<br>Double Deep Q-Learning RL"]
  B --> C["Data & Framework<br>Almgren-Chriss Model<br>Stochastic Impact Parameters"]
  C --> D["Computational Process<br>Neural Network Training<br>Simulation & Learning"]
  D --> E["Key Findings<br>1. Matches analytical solutions<br>2. Outperforms benchmarks<br>3. Handles latent liquidity"]

Cited References (79)

  1. Optimal execution in a time varying environment: well posedness and price manipulation (2024)
  2. Optimal signal-adaptive trading with temporary and transient price impact (2020) — link
  3. Optimal trading with stochastic liquidity and volatility (2012)
  4. Optimal execution strategies in limit order books with general shape functions (2010)
  5. Optimal trade execution and absence of price manipulations in limit order book models (2010)
  6. Stochastic duration and fast coupon bond option pricing in multi-factor models (1999)
  7. An exact bond option pricing formula (1989)
  8. Portfolio liquidation and ambiguity aversion (2018)
and 71 more
  1. Portfolio liquidation under factor uncertainty (2022)
  2. Optimal execution: A review (2022)
  3. Robust strategies for optimal order execution in the Almgren–Chriss framework (2013)
  4. V-shapes (2020)
  5. Optimal execution with non-linear transient market impact (2017)
  6. Transient linear price impact and Fredholm integral equations (2012)
  7. Optimal portfolio liquidation with limit orders (2012)
  8. Liquidation in limit order books with controlled intensity (2014)
  9. Optimal execution of portfolio transactions (2000)
  10. Optimal control of execution costs (1998)
  11. Optimal trading strategy and supply/demand dynamics (2013)
  12. The Pricing of Commodity Contracts. (1976)
  13. Theory of The Term Structure of Interest Rates (1985)
  14. Hedging under SABR model (2006)
  15. Fine-tune your smile: Correction to Hagan et al. (2008)
  16. The pricing of options and corporate liabilities (1973)
  17. An equilibrium characterization of the term structure (1977)
  18. Everything You Always Wanted to Know About Multiple Interest Rate Curve Bootstrapping but Were Afraid to Ask (2016)
  19. Impact of multiple curve dynamics in credit valuation adjustments under collateralization (2018)
  20. Price Impact on Term Structure (2020)
  21. Optimal dealer pricing under transaction and return uncertainty (1981)
  22. The dynamics of dealer markets under competition (1983)
  23. Inferring the components of the bid ask spread (1989)
  24. Bid, ask and transaction prices in a specialist market with heterogeneously informed traders (1985)
  25. Optimal portfolio liquidation with execution cost and risk (2010)
  26. Interest rates and the credit crunch: new formulas and market models (2009)
  27. Incorporating signals into optimal trading (2019)
  28. Interest-rate modelling in collateralized markets: multiple curves, credit-liquidity effects, CCPs (2013) — link
  29. Moving from IBORs to Alternative Risk Free Rates (2020)
  30. Almost-sure hedging with permanent price impact (2016)
  31. Pricing and hedging contingent claims with liquidity costs and market impact (2013)
  32. Optimal trade execution under displaced diffusions dynamics across different risk criteria (2014)
  33. Optimal trade execution: a mean quadratic variation approach (2012)
  34. Comparison between the mean-variance optimal and the mean-quadratic-variation optimal trading strategies (2013)
  35. General intensity shapes in optimal liquidation (2015)
  36. Incorporating order-flow into optimal execution (2016)
  37. Trading algorithms with learning in latent alpha models (2019)
  38. Reinforcement learning for optimized trade execution (2006)
  39. A reinforcement learning extension to the Almgren-Chriss framework for optimal trade execution (2014)
  40. Q-learning (1992)
  41. Playing atari with deep reinforcement learning (2013) — link
  42. Deep reinforcement learning with double q-learning (2016)
  43. Double deep q-learning for optimal execution (2021)
  44. Improving financial trading decisions using deep Q-learning: Predicting the number of shares, action strategies, and transfer learning (2019)
  45. Deep execution-value and policy based reinforcement learning for trading and beating market benchmarks (2019)
  46. Reinforcement learning for quantitative trading (2023)
  47. Recent advances in reinforcement learning in finance (2021) — link
  48. No-dynamic-arbitrage and market impact (2010)
  49. Price manipulation and quasi-arbitrage (2004)
  50. Adaptive market making via online learning (2013)
  51. Deep reinforcement learning for market making (2020)
  52. Model-based deep reinforcement learning for dynamic portfolio optimization (2019) — link
  53. Machine learning for market microstructure and high frequency trading (2013)
  54. Model-based reinforcement learning for predictions and control for limit order books (2019) — link
  55. Deep reinforcement learning for active high frequency trading (2021) — link
  56. Multi-agent reinforcement learning in a realistic limit order book market simulation (2020)
  57. Deep reinforcement learning for the optimal placement of cryptocurrency limit orders (2022)
  58. Continuous auctions and insider trading (1985)
  59. Deep differentiable reinforcement learning and optimal trading (2022)
  60. Optimal trading strategy and supply/demand dynamics (2013)
  61. Optimal trading with signals and stochastic price impact (2022)
  62. Liquidity fluctuations and the latent dynamics of price impact (2022)
  63. Optimal control of execution costs (1998)
  64. Direct estimation of equity market impact (2005)
  65. Fluctuations and response in financial markets: the subtle nature ofrandom’price changes (2003)
  66. How markets slowly digest changes in supply and demand (2009)
  67. Optimal basket liquidation for CARA investors is deterministic (2010)
  68. Algorithmic and high-frequency trading (2015)
  69. Reinforcement learning: an introduction MIT Press (1998)
  70. Optimal liquidation under stochastic price impact (2019)
  71. Measuring price impact and information content of trades in a time-varying setting — → 2212.12687

1 of 79 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.