Paper: arXiv 2610.03598

Authors: Siu Tung Wong, Carlo Campajola

Abstract

Reinforcement learning (RL) is increasingly used for financial optimal-control problems when complex dynamics make analytical strategies difficult to obtain. There are financial mathematics literactures which provides many solved models whose equations and controls could evaluate and guide learning; we ask whether RL can exploit these results. We place a proximal policy optimisation (PPO) agent in an analytically solved continuous-time broker–trader game. PPO replaces the broker and chooses its trading speed while interacting with an informed trader and stochastic uninformed order flow. We derive a finite-step reward from the broker’s continuous-time payoff and verify its discrete implementation through grid refinement and an exact one-step identity. With zero uninformed flow, a validation-selected PPO–FFNN approaches the reference action. With stochastic uninformed flow, the tested PPO–FFNN and PPO–LSTM remain inaccurate, although supervised learning confirms that their actors can represent the action. Monte Carlo diagnostics show that their critics do not reliably rank nearby actions; potential-based reward shaping also gives no reliable improvement. Under partial information, a causal certainty-equivalent controller based on the broker’s observable history remains close to the reference, while PPO has larger errors and lower payoffs. Finally, we freeze the analytical policy and train PPO to adjust it after the execution cost changes. Halving the cost yields a repeatable improvement that closes \(2.22\%\) of the gap to the changed-cost reference. The analytical solution therefore provides both a benchmark for diagnosing RL and a useful starting policy for adaptation.

Complexity vs Empirical Score

  • Math Complexity: 7.5/10
  • Empirical Rigor: 7.0/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: This paper demonstrates a strong blend of advanced mathematical modeling for the broker-trader game and rigorous empirical testing of RL agents against analytical benchmarks. The novelty lies in using analytically solved models to diagnose and guide PPO, offering a unique perspective on RL’s application in finance.

Research Flowchart

  flowchart TD
    A[Research Goal: Can RL exploit solved financial models?] --> B{Methodology: PPO in Broker-Trader Game};
    B --> C[Data/Inputs: Analytically Solved Continuous-Time Broker-Trader Game, Stochastic Uninformed Order Flow];
    C --> D{Computational Processes: PPO-FFNN, PPO-LSTM, Monte Carlo Diagnostics, Supervised Learning, Causal Certainty-Equivalent Controller};
    D --> E[Outcomes: PPO Inaccuracy with Stochastic Flow; Critic Fails to Rank Actions; CE Controller Outperforms PPO; PPO Adapts to Cost Changes];

Cited References (43)

  1. Nash Equilibrium between Brokers and Traders — → 2407.10561
  2. Incorporating Order-Flow into Optimal Execution (2016) — link
  3. Trading Algorithms with Learning in Latent Alpha Models (2019) — link
  4. Optimal Liquidation under Partial Information with Price Impact (2020) — link
  5. Broker-Trader Partial Information Nash-Equilibria (2024) — link
  6. Memory-Induced Supra-Competitive Outcomes Between Deep Reinforcement Learning Agents in Optimal Trade Execution (2026) — link
  7. Strategic Learning and Trading in Broker-Mediated Markets (2024) — link
  8. Deep Reinforcement Learning for Optimal Trading with Partial Information (2025) — link
and 35 more
  1. Double Deep Q-Learning for Optimal Execution (2021) — link
  2. Recurrent Networks, Hidden States and Beliefs in Partially Observable Environments (2022) — link
  3. Deep Variational Reinforcement Learning for POMDPs (2018) — link
  4. Unbiased Asymmetric Reinforcement Learning under Partial Observability (2022) — link
  5. Proximal Policy Optimization Algorithms (2017) — link
  6. An Algorithmic Introduction to Numerical Simulation of Stochastic Differential Equations (2001) — link
  7. A Reinforcement Learning Approach to Optimal Execution (2022)
  8. An End-to-End Optimal Trade Execution Framework based on Proximal Policy Optimization (2020) — link
  9. Reinforcement Learning for Optimized Trade Execution (2006) — link
  10. What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study (2020) — link
  11. Learn Continuously, Act Discretely: Hybrid Action-Space Reinforcement Learning For Optimal Execution (2022) — link
  12. Memory Gym: Towards Endless Tasks to Benchmark Memory Capabilities of Agents (2025) — link
  13. Towards Generalizable Reinforcement Learning for Trade Execution — → 2307.11685
  14. Towards Unified Attribution in Explainable AI , Data-Centric AI , and Mechanistic Interpretability (2025) — link
  15. Reinforcement Learning for Trade Execution with Market and Limit Orders — → 2507.06345
  16. Reinforcement Learning for Continuous-Time Optimal Execution: Actor-Critic Algorithm and Error Analysis (2026) — link
  17. Certainty Equivalence Is Efficient for Linear Quadratic Control (2019) — link
  18. Designing Rewards for Fast Learning (2022) — link
  19. Hyperparameters in Reinforcement Learning and How To Tune Them (2023) — link
  20. Implementation Matters in Deep RL : A Case Study on PPO and TRPO (2020) — link
  21. On Learning Intrinsic Rewards for Policy Gradient Methods (2018) — link
  22. Policy Invariance under Reward Transformations for General-Sum Stochastic Games (2011) — link
  23. Policy Invariance under Reward Transformations: Theory and Application to Reward Shaping (1999) — link
  24. RUDDER : Return Decomposition for Delayed Rewards (2019) — link
  25. Physics-Informed Neural Networks: A Deep Learning Framework for Solving Forward and Inverse Problems Involving Nonlinear Partial Differential Equations (2019) — link
  26. Solving High-Dimensional Partial Differential Equations Using Deep Learning (2018) — link
  27. A Reinforcement Learning Extension to the Almgren–Chriss Framework for Optimal Trade Execution (2014) — link
  28. Bridging Physics-Informed Neural Networks with Reinforcement Learning: Hamilton–Jacobi–Bellman Proximal Policy Optimization ( HJBPPO ) (2023) — link
  29. Data-Driven Initialization of Deep Learning Solvers for Hamilton–Jacobi–Bellman PDEs (2022) — link
  30. Residual Reinforcement Learning for Robot Control (2019) — link
  31. Warm-Start Actor–Critic: From Approximation Error to Sub-Optimality Gap (2023) — link
  32. Reinforcement Learning for Optimal Execution when Liquidity is Time-Varying (2024) — link
  33. Understanding Intermediate Layers Using Linear Classifier Probes (2017) — link
  34. Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator (2018) — link
  35. Deep Reinforcement Learning That Matters (2018) — link

3 of 43 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.