Paper: arXiv 2407.21025

Abstract

This paper establishes a new and comprehensive theoretical analysis for the application of reinforcement learning (RL) in high-frequency market making. We bridge the modern RL theory and the continuous-time statistical models in high-frequency financial economics. Different with most existing literature on methodological research about developing various RL methods for market making problem, our work is a pilot to provide the theoretical analysis. We target the effects of sampling frequency, and find an interesting tradeoff between error and complexity of RL algorithm when tweaking the values of the time increment $Δ$ $-$ as $Δ$ becomes smaller, the error will be smaller but the complexity will be larger. We also study the two-player case under the general-sum game framework and establish the convergence of Nash equilibrium to the continuous-time game equilibrium as $Δ\rightarrow0$. The Nash Q-learning algorithm, which is an online multi-agent RL method, is applied to solve the equilibrium. Our theories are not only useful for practitioners to choose the sampling frequency, but also very general and applicable to other high-frequency financial decision making problems, e.g., optimal executions, as long as the time-discretization of a continuous-time markov decision process is adopted. Monte Carlo simulation evidence support all of our theories.

Complexity vs Empirical Score

  • Math Complexity: 9.0/10
  • Empirical Rigor: 6.0/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper is highly theoretical with dense mathematical analysis of continuous-time models, convergence, and sample complexity, yet it includes Monte Carlo simulations to support the theories, indicating moderate empirical rigor.

Research Flowchart

  flowchart TD
  A["Research Goal<br/>Establish theoretical analysis for RL<br/>in high-frequency market making"] --> B["Methodology: Continuous-Time &<br/>Game Theory Framework"]
  B --> C["Key Analysis: Effect of Sampling Frequency Δ<br/>Tradeoff: Error ↓ vs Complexity ↑ as Δ→0"]
  B --> D["Multi-Agent Extension<br/>General-Sum Game & Nash Q-Learning"]
  C & D --> E["Computational Process<br/>Monte Carlo Simulation"]
  E --> F["Key Outcomes & Findings"]
  F --> G["Convergence Proof<br/>Nash Eq → Continuous-Time Eq as Δ→0"]
  F --> H["Practical Guidance<br/>Optimal Δ selection for practitioners"]
  F --> I["General Applicability<br/>Extends to optimal execution & other HF problems"]

Cited References (41)

  1. Robust Market Making: To Quote, or not To Quote — → 2508.16588
  2. Dynamics of market making algorithms in dealer markets: Learning and tacit collusion (2023) — link
  3. High frequency market making: The role of speed (2023) — link
  4. Can maker-taker fees prevent algorithmic cooperation in market making? (2022) — link
  5. AI-driven liquidity provision in OTC financial markets (2022) — link
  6. Towards multi‐agent reinforcement learning‐driven over‐the‐counter market simulations (2022) — link
  7. Approximate Q Learning for Controlled Diffusion Processes and Its Near Optimality (2022) — link
  8. Recent advances in reinforcement learning in finance (2021) — link
and 33 more
  1. Interactions of market making algorithms: a study on perceived collusion (2021) — link
  2. Reinforcement Learning Approaches to Optimal Market Making (2021) — link
  3. Towards a fully rl-based market simulator (2021) — link
  4. Dynamic Equilibrium of Market Making with Price Competition (2020) — link
  5. Reinforcement Learning for Market Making in a Multi-agent Dealer Market (2019) — link
  6. Financial Machine Learning (2019) — link
  7. Q-learning with Nearest Neighbors (2018) — link
  8. High-Frequency Financial Econometrics (2016) — link
  9. Buy Low, Sell High: A High Frequency Trading Perspective (2014) — link
  10. Dealing with the inventory risk: a solution to the market making problem (2011) — link
  11. A Stochastic Model for Order Book Dynamics (2010) — link
  12. High-frequency trading in a limit order book (2008) — link
  13. Applied stochastic processes and control for jump-diffusions - modeling, analysis, and computation (2007)
  14. Games with Imperfectly Observable Actions in Continuous Time (2007) — link
  15. Nonzero-sum games for continuous-time Markov chains with unbounded discounted payoffs (2005) — link
  16. Learning Rates for Q-learning (2004) — link
  17. Nash Q-Learning for General-Sum Stochastic Games (2003)
  18. Competitive Markov Decision Processes (1996) — link
  19. Controlled markov processes and viscosity solutions (1994) — link
  20. Towards a Theory of Discounted Repeated Games with Imperfect Monitoring (1990) — link
  21. Informed Speculation with Imperfect Competition (1989) — link
  22. Continuous Auctions and Insider Trading (1985) — link
  23. On the Convergence of the Discrete Time Dynamic Programming Equation for General Semigroups (1982) — link
  24. Optimal dealer pricing under transactions and return uncertainty (1981) — link
  25. Dealership market: Market-making with inventory (1980) — link
  26. Continuously Discounted Markov Decision Model with Countable State and Action Space (1971) — link
  27. Algorithmic Collusion in Electronic Markets: The Impact of Tick Size (2022) — link
  28. Sample Complexity Bounds of Exploration (2012) — link
  29. Reinforcement Learning: An Introduction (2005) — link
  30. Non-Cooperative Games (2003)
  31. Market structure, information, futures markets, and price formation. (1984)
  32. Controlled Stochastic Processes (1979) — link
  33. Assumption 2

1 of 41 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.