Paper: arXiv 2509.12456

Authors: Rafael Zimmer, Oswaldo Luiz do Valle Costa

Abstract

Reinforcement Learning has emerged as a promising framework for developing adaptive and data-driven strategies, enabling market makers to optimize decision-making policies based on interactions with the limit order book environment. This paper explores the integration of a reinforcement learning agent in a market-making context, where the underlying market dynamics have been explicitly modeled to capture observed stylized facts of real markets, including clustered order arrival times, non-stationary spreads and return drifts, stochastic order quantities and price volatility. These mechanisms aim to enhance stability of the resulting control agent, and serve to incorporate domain-specific knowledge into the agent policy learning process. Our contributions include a practical implementation of a market making agent based on the Proximal-Policy Optimization (PPO) algorithm, alongside a comparative evaluation of the agent’s performance under varying market conditions via a simulator-based environment. As evidenced by our analysis of the financial return and risk metrics when compared to a closed-form optimal solution, our results suggest that the reinforcement learning agent can effectively be used under non-stationary market conditions, and that the proposed simulator-based environment can serve as a valuable tool for training and pre-training reinforcement learning agents in market-making scenarios.

Complexity vs Empirical Score

  • Math Complexity: 8.5/10
  • Empirical Rigor: 6.0/10 (capped from 7.0: no out-of-sample test; see penalty flags)
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper employs advanced stochastic control and Markov Decision Process formalism (high math) and implements a practical PPO agent in a custom simulator with financial risk metrics (high rigor).

Research Flowchart

  flowchart TD
  A["Research Goal:<br>Adaptive Market Making<br>via RL in Non-Stationary LOBs"] --> B["Methodology:<br>Proximal-Policy Optimization (PPO) Agent"]
  B --> C["Simulator Environment<br>Inputs: Stylized Facts<br>(Clustered arrivals, Volatility, Drifts)"]
  C --> D["Computational Process:<br>Agent-Environment Interaction Loop"]
  D --> E["Comparative Evaluation:<br>RL Agent vs. Closed-Form Optimal Solution"]
  E --> F["Key Findings:<br>Effective Non-Stationary Adaptation<br>Validated Simulator for Pre-training"]

Cited References (84)

  1. ZeroSwap: Data-driven Optimal Market Making in DeFi (2023) — link
  2. IMM: An Imitative Reinforcement Learning Approach with Predictive Representation Learning for Automatic Market Making (2023) — link
  3. Over-the-Counter Market Making via Reinforcement Learning — → 2307.01816
  4. Market Making of Options via Reinforcement Learning (2023) — link
  5. Integrating Tick-level Data and Periodical Signal for High-frequency Market Making — → 2306.17179
  6. Systematic Review on Reinforcement Learning in the Field of Fintech — → 2305.07466
  7. Predictive Crypto-Asset Automated Market Making Architecture for Decentralized Finance using Deep Reinforcement Learning (2022) — link
  8. Model-based gym environments for limit order book trading (2022) — link
and 76 more
  1. Market Making via Reinforcement Learning in China Commodity Market (2022) — link
  2. Improved Method of Stock Trading under Reinforcement Learning Based on DRQN and Sentiment Indicators ARBR (2021) — link
  3. Using Echo State Networks to Approximate Value Functions for Control (2021) — link
  4. Extending Deep Reinforcement Learning Frameworks in Cryptocurrency Market Making (2020) — link
  5. Deep Reinforcement Learning in Cryptocurrency Market Making (2019) — link
  6. Deep reinforcement learning for market making in corporate bonds: beating the curse of dimensionality (2019) — link
  7. Optimizing Market Making using Multi-Agent Reinforcement Learning (2018) — link
  8. Closed-form approximations in multi-asset market making (2018) — link
  9. Market Making via Reinforcement Learning (2018) — link
  10. Can maker-taker fees prevent algorithmic cooperation in market making? (2022) — link
  11. Effect of Market Spread Over Reinforcement Learning Based Market Maker (2019) — link
  12. Reinforcement Learning Approaches to Optimal Market Making (2021) — link
  13. Market Making With Signals Through Deep Reinforcement Learning (2021) — link
  14. Automated market maker inventory management with deep reinforcement learning (2023) — link
  15. Market Making Strategy Optimization via Deep Reinforcement Learning (2022) — link
  16. Multi-Asset Market Making via Multi-Task Deep Reinforcement Learning (2022) — link
  17. Market Making with Deep Reinforcement Learning from Limit Order Books — → 2305.15821
  18. Deep Reinforcement Learning for Market Making Under a Hawkes Process-Based Limit Order Book Model (2022) — link
  19. Algorithmic market making for options (2021) — link
  20. Robust Market Making via Adversarial Reinforcement Learning (2020) — link
  21. FairMM: A Fast and Frontrunning-Resistant Crypto Market-Maker (2022) — link
  22. Optimal Market Making with Persistent Order Flow (2021) — link
  23. Market Making and Incentives Design in the Presence of a Dark Pool: A Stackelberg Actor-Critic Approach (2022) — link
  24. Dealer markets: A reinforcement learning mean field game approach (2023) — link
  25. Profit equitably: An investigation of market maker’s impact on equitable outcomes (2021) — link
  26. Optimal Trading on a Dynamic Curve Automated Market Maker (2022) — link
  27. Interactions of Market Making Algorithms: a Study on Perceived Collusion (2021) — link
  28. A reinforcement learning approach to improve the performance of the Avellaneda-Stoikov market-making algorithm (2022) — link
  29. Computational Methods for Market Making Algorithms (2022) — link
  30. Dynamics of market making algorithms in dealer markets: Learning and tacit collusion (2023) — link
  31. Forex market forecasting using machine learning: Systematic Literature Review and meta-analysis (2023) — link
  32. Recent advances in reinforcement learning in finance (2023) — link
  33. Algorithmic market making in dealer markets with hedging and market impact (2023) — link
  34. Optimal Liquidation, Acquisition and Market Making Problems in HFT under Hawkes Models for LOB (2022) — link
  35. Deep Q-Learning Market Makers in a Multi-Agent Simulated Stock Market (2021) — link
  36. Size matters for OTC market makers: General results and dimensionality reduction techniques (2021) — link
  37. Cooperation between independent market makers (2022) — link
  38. Empirical Analysis on Market-Making Hedging Problem based on Quadratic Programming with Time-Decay and Sparse-Term (2022)
  39. AI-driven liquidity provision in OTC financial markets (2022) — link
  40. STBM: Stochastic Trading Behavior Model for Financial Markets (2021) — link
  41. Towards a fully RL-based Market Simulator (2021) — link
  42. Algorithmic trading in a microstructural limit order book model (2020) — link
  43. Investigating the role of utilitarian and hedonic goals in characterizing customer loyalty in E-marketplaces (2023) — link
  44. Approximation algorithms for the selling with preference (2020) — link
  45. A CLOSED-FORM SOLUTION FOR OPTIMAL ORNSTEIN-UHLENBECK DRIVEN TRADING STRATEGIES (2020) — link
  46. Beating the market with a bad predictive model (2023) — link
  47. A logarithmic market scoring rule agent-based model to evaluate prediction markets (2023) — link
  48. Novel Deep Learning Model with CNN and Bi-Directional LSTM for Improved Stock Market Index Prediction (2019) — link
  49. Predicting Stock Price Changes Based on the Limit Order Book: A Survey (2022) — link
  50. Temporal Difference Learning and TD-Gammon (1995) — link
  51. Reinforcement Learning: A Survey (1996) — link
  52. Reinforcement Learning: An Introduction (2018) — link
  53. Reinforcement Learning for Market Making in a Multi-agent Dealer Market (2019) — link
  54. Optimal market making (2017) — link
  55. High-frequency trading in a limit order book (2008) — link
  56. Optimal Market Making by Reinforcement Learning (2021) — link
  57. Dealing with the inventory risk: a solution to the market making problem (2012) — link
  58. Market-making with reinforcement-learning (SAC) (2020) — link
  59. ABIDES: Towards High-Fidelity Multi-Agent Market Simulation (2020) — link
  60. Simulation and estimation of an agent-based market-model with a matching engine (2021) — link
  61. Dealing with the inventory risk: a solution to the market making problem (2012) — link
  62. Algorithmic and High-Frequency Trading (2015) — link
  63. Market Microstructure Theory (1995)
  64. Bid, Ask, and Transaction Prices in a Specialist Market with Heterogeneously Informed Traders (1985)
  65. Aspects of the Theory of Risk-Bearing (1965)
  66. Risk Aversion in the Small and in the Large (1964) — link
  67. Proximal Policy Optimization Algorithms (2017) — link
  68. IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures (2018) — link
  69. High-Dimensional Continuous Control Using Generalized Advantage Estimation (2015) — link
  70. Network Structure and Systemic Risk in Banking Systems (2010) — link
  71. Trades, Quotes and Prices: Financial Markets Under the Microscope (2018)
  72. JAX-LOB: A GPU-Accelerated limit order book simulator to unlock large scale reinforcement learning for trading (2023) — link
  73. Human-level control through deep reinforcement learning (2015) — link
  74. Deep Reinforcement Learning with Double Q-learning (2015) — link
  75. On the Theory of the Brownian Motion (1930) — link
  76. A Theory of the Term Structure of Interest Rates (1985) — link

4 of 84 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.