Paper: arXiv 2410.14504

Abstract

We develop a deep reinforcement learning (RL) framework for an optimal market-making (MM) trading problem, specifically focusing on price processes with semi-Markov and Hawkes Jump-Diffusion dynamics. We begin by discussing the basics of RL and the deep RL framework used, where we deployed the state-of-the-art Soft Actor-Critic (SAC) algorithm for the deep learning part. The SAC algorithm is an off-policy entropy maximization algorithm more suitable for tackling complex, high-dimensional problems with continuous state and action spaces like in optimal market-making (MM). We introduce the optimal MM problem considered, where we detail all the deterministic and stochastic processes that go into setting up an environment for simulating this strategy. Here we also give an in-depth overview of the jump-diffusion pricing dynamics used, our method for dealing with adverse selection within the limit order book, and we highlight the working parts of our optimization problem. Next, we discuss training and testing results, where we give visuals of how important deterministic and stochastic processes such as the bid/ask, trade executions, inventory, and the reward function evolved. We include a discussion on the limitations of these results, which are important points to note for most diffusion models in this setting.

Complexity vs Empirical Score

  • Math Complexity: 8.5/10
  • Empirical Rigor: 4.0/10
  • Quadrant: Lab Rats — theoretically deep, empirically untested

Why this score: The paper employs advanced stochastic calculus and deep reinforcement learning theory, including complex non-Markovian processes like Hawkes jump-diffusions, which drives a high math score. However, the empirical section primarily discusses simulated training and testing results with noted limitations, lacking the backtest-ready, real-market implementation details or statistical metrics typical of high-rigor work.

Research Flowchart

  flowchart TD
  A["Research Goal:<br>Optimal Market-Making<br>in Non-Markov Dynamics"] --> B["Methodology:<br>Deep RL using<br>Soft Actor-Critic SAC"]
  B --> C["Input Data:<br>Hawkes Jump-Diffusion<br>Limit Order Book"]
  C --> D["Computational Process:<br>Training SAC on<br>Stochastic Environment"]
  D --> E["Key Findings:<br>Optimal Inventory &<br>Spread Management<br>Limitations Identified"]

Cited References (77)

  1. Optimal execution of portfolio transactions (2001)
  2. A discussion paper for possible approaches to building a statistically valid backtesting framework (2024)
  3. Deep attentive survival analysis in limit order books: Estimating fill probabilities with convolutional-transformers (2024)
  4. Deep reinforcement learning: A brief survey (2017)
  5. High-frequency trading in a limit order book (2008)
  6. Hawkes processes in finance (2015)
  7. Market making and incentives design in the presence of a dark pool: a Stackelberg actor–critic approach (2023)
  8. The exploration-exploitation dilemma: a multidisciplinary framework (2014)
and 69 more
  1. Optimal control of execution costs (1998)
  2. Optimal control of trading algorithms: a general impulse control approach (2011)
  3. Price discovery without trading: Evidence from limit orders (2019)
  4. Optimal execution of limit and market orders with trade director, speed limiter, and fill uncertainty (2017)
  5. Enhancing trading strategies with order book signals (2018)
  6. Algorithmic and high-frequency trading (2015)
  7. Buy low, sell high: A high frequency trading perspective (2014)
  8. Algorithmic trading, stochastic control, and mutually exciting processes (2018)
  9. Deep reinforcement learning for algorithmic trading (2021)
  10. An electronic market-maker (2001)
  11. The price impact of order book events (2014)
  12. Price dynamics in a Markovian limit order market (2013)
  13. Portfolio selection with transaction costs (1990)
  14. Limit Order Book ( LOB ) shape modeling in presence of heterogeneously informed market participants (2019)
  15. The negative drift of a limit order fill — → 2407.16527
  16. High-Frequency Trading (2016) — link
  17. Flow toxicity and liquidity in a high-frequency world (2012)
  18. Semi- Markov model for market microstructure (2015)
  19. High frequency trading and asymptotics for small risk aversion in a Markov renewal model (2015)
  20. Taming the noise in reinforcement learning via soft updates (2015) — link
  21. Reinforcement learning approaches to optimal market making (2021)
  22. Deep reinforcement learning for market making under a Hawkes process-based limit order book model (2022)
  23. Limit order books (2013)
  24. Optimal market making (2017)
  25. Deep reinforcement learning for market making in corporate bonds: beating the curse of dimensionality (2019)
  26. Spectra of some self-exciting and mutually exciting point processes (1971)
  27. Reinforcement learning with deep energy-based policies (2017)
  28. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor (2018)
  29. A cluster process representation of a self-exciting process (1974)
  30. Quantitative and comparative analyses of limit order books with general compound Hawkes processes (2019)
  31. A dynamic limit order market with fast and slow traders (2014)
  32. Market Making at the Touch with Short-Term Alpha (Chap 10.4.2) (2019)
  33. Limit Order Book Simulations: A Review — → 2402.17359
  34. Algorithmic and High-Frequency Trading Problems for Semi- Markov and Hawkes Jump-Diffusion Models — → 2409.12776
  35. Market Simulation under Adverse Selection — → 2409.12721
  36. Market making under a weakly consistent limit order book model (2019)
  37. Deep robust reinforcement learning for practical algorithmic trading (2019)
  38. Fill Probabilities in a Limit Order Book with State-Dependent Stochastic Order Flows — → 2403.02572
  39. A deep learning approach to estimating fill probabilities in a limit order book (2022)
  40. Forecasting jump arrivals in stock prices: new attention-based network architecture using limit order book data (2019)
  41. Lifetime portfolio selection under uncertainty: The continuous-time case (1969)
  42. Human-level control through deep reinforcement learning (2015)
  43. Using full Limit Order Book for price jump prediction (2020)
  44. Optimal consumption and portfolio with both fixed and proportional transaction costs (2002)
  45. Continuous-time stochastic control and optimization with financial applications (2009)
  46. Deep reinforcement learning in quantitative algorithmic trading: A review (2021) — link
  47. Sequence level training with recurrent neural networks (2015) — link
  48. On stochastic optimal control and reinforcement learning by approximate inference (2013)
  49. Optimal Liquidation, Acquisition and Market Making Problems in HFT under Hawkes Models for LOB (2022)
  50. Stochastic Optimal Control Problems in LOB for Various Stochastic Models (2023)
  51. Deep reinforcement learning in cryptocurrency market making (2019) — link
  52. Extending deep reinforcement learning frameworks in cryptocurrency market making (2020) — link
  53. LOB modeling using Hawkes processes with a state-dependent factor (2021) — link
  54. Mastering the game of Go with deep neural networks and tree search (2016)
  55. How are reinforcement learning and deep learning algorithms used for big data based decision making in financial industries–A review and research agenda (2022)
  56. General Compound Hawkes Processes for Mid-Price Prediction (2021) — link
  57. 79+ amazing algorithmic trading statistics (2016) — link
  58. Stable Baselines3 (2020) — link
  59. A semi- Markovian modeling of limit order markets (2017)
  60. General semi- Markov model for limit order books (2017)
  61. General compound Hawkes processes in limit order books (2020)
  62. Compound Hawkes processes in limit order books (2019)
  63. Financial modelling with jump processes (2003)
  64. An application of deep reinforcement learning to algorithmic trading (2021)
  65. Robot trajectory optimization using approximate inference (2009)
  66. Optimal investment-consumption models with constraints (1989)
  67. Statistical Analysis of the Price Jumps of Financial Assets Based on LOB Data (2017)
  68. Deep reinforcement learning for trading (2019) — link
  69. Maximum entropy inverse reinforcement learning. (2008)

5 of 77 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.