MacroHFT: Memory Augmented Context-aware Reinforcement Learning On High Frequency Trading

Paper: arXiv 2406.14537

Abstract

High-frequency trading (HFT) that executes algorithmic trading in short time scales, has recently occupied the majority of cryptocurrency market. Besides traditional quantitative trading methods, reinforcement learning (RL) has become another appealing approach for HFT due to its terrific ability of handling high-dimensional financial data and solving sophisticated sequential decision-making problems, \emph{“e.g.,”} hierarchical reinforcement learning (HRL) has shown its promising performance on second-level HFT by training a router to select only one sub-agent from the agent pool to execute the current transaction. However, existing RL methods for HFT still have some defects: 1) standard RL-based trading agents suffer from the overfitting issue, preventing them from making effective policy adjustments based on financial context; 2) due to the rapid changes in market conditions, investment decisions made by an individual agent are usually one-sided and highly biased, which might lead to significant loss in extreme markets. To tackle these problems, we propose a novel Memory Augmented Context-aware Reinforcement learning method On HFT, \emph{“a.k.a.”} MacroHFT, which consists of two training phases: 1) we first train multiple types of sub-agents with the market data decomposed according to various financial indicators, specifically market trend and volatility, where each agent owns a conditional adapter to adjust its trading policy according to market conditions; 2) then we train a hyper-agent to mix the decisions from these sub-agents and output a consistently profitable meta-policy to handle rapid market fluctuations, equipped with a memory mechanism to enhance the capability of decision-making. Extensive experiments on various cryptocurrency markets demonstrate that MacroHFT can achieve state-of-the-art performance on minute-level trading tasks.

Complexity vs Empirical Score

• Math Complexity: 6.0/10
• Empirical Rigor: 6.0/10 (capped from 7.0: transaction costs only partially modelled; see penalty flags)
• Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper employs advanced reinforcement learning concepts (HRL, conditional adapters, memory mechanisms) and involves complex training phases, indicating high mathematical density. It provides extensive experiments on real cryptocurrency markets with minute-level data, includes code release, and compares against state-of-the-art baselines, demonstrating strong empirical rigor.

Research Flowchart

  flowchart TD
  Goal["Research Goal<br>Create context-aware RL for HFT<br>addressing overfitting & bias"] --> Method["MacroHFT Methodology<br>Two-Phase Training"]
  
  Data["Input: Cryptocurrency Market Data<br>Decomposed by Trends & Volatility"] --> Phase1["Phase 1: Sub-Agents Training<br>Multiple agents with<br>Conditional Adapters"]
  
  Phase1 --> Phase2["Phase 2: Hyper-Agent Training<br>Memory-Augmented Mixing<br>Meta-Policy Optimization"]
  
  Phase2 --> Findings["Key Findings<br>State-of-the-art performance on<br>minute-level HFT tasks"]
  
  style Goal fill:#e1f5e1
  style Findings fill:#fff3cd

Cited References (31)

1. Earnhft: Efficient hierarchical reinforcement learning for high frequency trading — → 2309.12891
2. Quantitative trading through random perturbation Q-network with nonlinear transaction costs (2022)
3. Towards generalizable reinforcement learning for trade execution — → 2307.11685
4. Dueling network architectures for deep reinforcement learning (2016)
5. Scalable diffusion models with transformers (2023)
6. A dynamic model of the limit order book (2009)
7. Market microstructure: A survey (2000)
8. Neural episodic control (2017)

3 of 31 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.