Reinforcement Learning Pair Trading: A Dynamic Scaling approach

Paper: arXiv 2407.16103

Abstract

Cryptocurrency is a cryptography-based digital asset with extremely volatile prices. Around USD 70 billion worth of cryptocurrency is traded daily on exchanges. Trading cryptocurrency is difficult due to the inherent volatility of the crypto market. This study investigates whether Reinforcement Learning (RL) can enhance decision-making in cryptocurrency algorithmic trading compared to traditional methods. In order to address this question, we combined reinforcement learning with a statistical arbitrage trading technique, pair trading, which exploits the price difference between statistically correlated assets. We constructed RL environments and trained RL agents to determine when and how to trade pairs of cryptocurrencies. We developed new reward shaping and observation/action spaces for reinforcement learning. We performed experiments with the developed reinforcement learner on pairs of BTC-GBP and BTC-EUR data separated by 1 min intervals (n=263,520). The traditional non-RL pair trading technique achieved an annualized profit of 8.33%, while the proposed RL-based pair trading technique achieved annualized profits from 9.94% to 31.53%, depending upon the RL learner. Our results show that RL can significantly outperform manual and traditional pair trading techniques when applied to volatile markets such as~cryptocurrencies.

Complexity vs Empirical Score

• Math Complexity: 6.5/10
• Empirical Rigor: 7.0/10
• Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper employs advanced mathematics including MDP formulation, reward shaping, and statistical arbitrage concepts, while demonstrating empirical rigor through backtesting on large-scale cryptocurrency datasets (263,520 data points) and reporting annualized profit metrics.

Research Flowchart

  flowchart TD
  A["Research Goal:<br>RL vs Traditional Methods<br>for Crypto Pair Trading"] --> B["Data Collection:<br>BTC-GBP & BTC-EUR Data<br>1-min intervals, n=263,520"]
  B --> C["Methodology:<br>Construct RL Environment<br>Define Reward/Action Spaces"]
  C --> D["Training:<br>Train RL Agents<br>(RL Learners)"]
  D --> E["Comparison:<br>RL vs Traditional<br>Pair Trading"]
  E --> F["Outcome:<br>RL Outperformance:<br>9.94% - 31.53% vs 8.33%"]
  style F fill:#d4edda,stroke:#155724

Cited References (175)

1. Deep Reinforcement Learning Strategies in Finance : Insights into Asset Holding , Trading Behavior , and Purchase Diversity (2024) — link
2. Deep Reinforcement Learning for Active High Frequency Trading (2023) — link
3. Shapify.me (2024) — link
4. Statistics: Fourth international student edition (2020)
5. Optimal Portfolio Management When Stocks Are Driven by Mean Reverting Processes (2019) — link
6. Generalized Stochastic Arbitrage Opportunities (2024) — link
7. Differential Evolution – A Simple and Efficient Heuristic for global Optimization over Continuous Spaces (1997) — link
8. Constructing cointegrated cryptocurrency portfolios for statistical arbitrage (2019) — link

2 of 175 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.