Paper: arXiv 2511.02016

Authors: Patrick Cheridito, Jean-Loup Dupret, Zhexin Wu

Abstract

We present ABIDES-MARL, a framework that combines a new multi-agent reinforcement learning (MARL) methodology with a new realistic limit-order-book (LOB) simulation system to study equilibrium behavior in complex financial market games. The system extends ABIDES-Gym by decoupling state collection from kernel interruption, enabling synchronized learning and decision-making for multiple adaptive agents while maintaining compatibility with standard RL libraries. It preserves key market features such as price-time priority and discrete tick sizes. Methodologically, we use MARL to approximate equilibrium-like behavior in multi-period trading games with a finite number of heterogeneous agents-an informed trader, a liquidity trader, noise traders, and competing market makers-all with individual price impacts. This setting bridges optimal execution and market microstructure by embedding the liquidity trader’s optimization problem within a strategic trading environment. We validate the approach by solving an extended Kyle model within the simulation system, recovering the gradual price discovery phenomenon. We then extend the analysis to a liquidity trader’s problem where market liquidity arises endogenously and show that, at equilibrium, execution strategies shape market-maker behavior and price dynamics. ABIDES-MARL provides a reproducible foundation for analyzing equilibrium and strategic adaptation in realistic markets and contributes toward building economically interpretable agentic AI systems for finance.

Complexity vs Empirical Score

  • Math Complexity: 7.0/10
  • Empirical Rigor: 6.0/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper introduces a complex multi-agent reinforcement learning framework and connects it to theoretical models like Kyle’s model, which involves substantial mathematical modeling and equilibrium concepts. However, while it presents a simulation system (ABIDES-MARL) with backtest-ready features like LOB simulation and agents, the empirical validation is primarily theoretical (recovering known models) rather than data-heavy backtesting on real market data.

Research Flowchart

  flowchart TD
  A["Research Goal<br>Model equilibrium behavior in<br>complex financial market games"] --> B["Methodology: MARL Framework"]
  B --> C["Core Components<br>Limit Order Book Simulation &<br>Synchronized Multi-Agent RL"]
  C --> D["Key Processes"]
  D --> E["Solve Extended Kyle Model"]
  D --> F["Simulate Endogenous Liquidity"]
  E --> G["Findings & Outcomes"]
  F --> G
  G --> H["Recovered Gradual Price Discovery"]
  G --> I["Equilibrium Strategies Shape<br>Market-Maker Behavior & Price Dynamics"]

Cited References (78)

  1. Algorithmic and High-Frequency Trading (2015)
  2. Optimal execution strategies in limit order books with general shape functions (2010)
  3. Decentralized policy optimization (2022) — link
  4. On stationary point convergence of PPO-Clip (2023)
  5. Ppo-clip attains global optimality: Towards deeper understandings of clipping (2024)
  6. Continuous auctions and insider trading (1985)
  7. Bid, ask and transaction prices in a specialist market with heterogeneously informed traders (1985) — link
  8. Liquidity and market structure (1988)
and 70 more
  1. Optimal dealer pricing under transactions and return uncertainty (1981) — link
  2. Long-lived private information and imperfect competition (1992)
  3. The effect of public information and competition on trading volume and price volatility (1993)
  4. Deep reinforcement learning with double Q-learning (2016)
  5. Proximal policy optimization algorithms (2017) — link
  6. Multi-agent actor-critic for mixed cooperative-competitive environments (2017) — link
  7. ABIDES: Towards high-fidelity multi-agent market simulation (2020) — link
  8. ABIDES-Gym: Gym environments for multi-agent discrete event simulation and application to financial markets (2021) — link
  9. Sample selection bias as a specification error (1979) — link
  10. Explanatory Model Analysis: Interpretability for Machine Learning Models (2021)
  11. Equilibrium fast trading (2015) — link
  12. Optimal execution of portfolio transactions (2001) — link
  13. Optimal control of execution costs (1998) — link
  14. Continuous control with deep reinforcement learning (2019) — link
  15. The surprising effectiveness of PPO in cooperative, multi-agent games (2022)
  16. Trust region policy optimisation in multi-agent reinforcement learning (2022) — link
  17. Multi-agent reinforcement learning in a realistic limit order book market simulation (2020) — link
  18. RLlib: Abstractions for distributed reinforcement learning (2018) — link
  19. MARLlib: A scalable and efficient multi-agent reinforcement learning library (2023)
  20. Q-learning (1992) — link
  21. Simple statistical gradient-following algorithms for connectionist reinforcement learning (1992) — link
  22. Deterministic policy gradient algorithms (2014) — link
  23. Reinforcement learning for optimized trade execution (2006)
  24. A reinforcement learning extension to the Almgren–Chriss framework for optimal trade execution (2014) — link
  25. Superhuman AI for multiplayer poker (2019) — link
  26. Learning to communicate with deep multi-agent reinforcement learning (2016) — link
  27. Deep investing in Kyle’s single period model (2020) — link
  28. High-frequency trading in a limit order book (2008) — link
  29. Reinforcement learning for market making in a multi-agent dealer market (2019) — link
  30. Gymnasium: A standard interface for reinforcement learning environments (2024) — link
  31. PettingZoo: Gym for multi-agent reinforcement learning (2021) — link
  32. Optimal execution with reinforcement learning (2024) — link
  33. Asynchronous deep double dueling Q-learning for trading-signal execution in limit order book markets (2023) — link
  34. Spoofing the limit order book: A strategic agent-based analysis (2021) — link
  35. Strategic trading when agents forecast the forecasts of others (1996)
  36. Optimal liquidity trading (2005) — link
  37. Episodic liquidity crises: Cooperative and predatory trading (2007) — link
  38. Market making via reinforcement learning (2018) — link
  39. Robust market making via adversarial reinforcement learning (2020) — link
  40. Price manipulation and quasi-arbitrage (2004)
  41. Market architecture: Limit-order books versus dealership markets (2002) — link
  42. Floors, dealer markets and limit order markets (1998) — link
  43. Market Liquidity: Theory, Evidence, and Policy (2023) — link
  44. Double deep Q-learning for optimal execution (2021) — link
  45. An end-to-end optimal trade execution framework based on proximal policy optimization (2020) — link
  46. Informed speculation with imperfect competition (1989) — link
  47. Stable-Baselines3: Reliable reinforcement learning implementations (2021)
  48. An agent-based market simulator for back-testing deep reinforcement learning based trade execution strategies (2021)
  49. Robustness of equilibrium in the Kyle model of informed speculation (2015) — link
  50. Robust strategies for optimal order execution in the Almgren–Chriss framework (2013) — link
  51. From deep learning to LLMs: A survey of AI in quantitative investment (2025) — link
  52. FinMem: A performance-enhanced LLM trading agent with layered memory and character design (2023) — link
  53. A multimodal foundation agent for financial trading: Tool-augmented, diversified, and generalist (2024) — link
  54. FinRobot: An open-source AI agent platform for financial applications using large language models (2024) — link
  55. Chain-of-thought is not explainability (2025) — link
  56. Deep differentiable reinforcement learning and optimal trading (2022) — link
  57. Do price trajectory data increase the efficiency of market impact estimation? (2024) — link
  58. Optimal liquidation under indirect price impact with propagator (2025) — link
  59. A reinforcement learning approach to optimal execution (2022) — link
  60. No-dynamic-arbitrage and market impact (2010) — link
  61. Optimal trading strategy and supply/demand dynamics (2013) — link
  62. Optimal execution with non-linear transient market impact (2017) — link
  63. Autoregressive conditional heteroscedasticity with estimates of the variance of United Kingdom inflation (1982) — link
  64. statsmodels: Econometric and statistical modeling with Python (2010)
  65. Mean field games (2007) — link
  66. Large population stochastic dynamic games: Closed-loop McKean–Vlasov systems and the Nash certainty equivalence principle (2006) — link
  67. Reinforcement learning for trade execution with market impact (2025) — link
  68. Reinforcement learning in queue-reactive models: application to optimal execution (2025) — link
  69. Agentic Artificial Intelligence in Finance: A Comprehensive Survey (2025) — link
  70. Optimal trade execution under stochastic volatility and liquidity (2014) — link

0 of 78 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.