Trade-R1: Bridging Verifiable Rewards to Stochastic Environments via Process-Level Reasoning Verification

Paper: arXiv 2601.03948

Authors: Rui Sun, Yifan Sun, Sheng Xu, Li Zhao, Jing Li, Daxin Jiang, Cheng Hua, Zuo Bai

Abstract

Reinforcement Learning (RL) has enabled Large Language Models (LLMs) to achieve remarkable reasoning in domains like mathematics and coding, where verifiable rewards provide clear signals. However, extending this paradigm to financial decision is challenged by the market’s stochastic nature: rewards are verifiable but inherently noisy, causing standard RL to degenerate into reward hacking. To address this, we propose Trade-R1, a model training framework that bridges verifiable rewards to stochastic environments via process-level reasoning verification. Our key innovation is a verification method that transforms the problem of evaluating reasoning over lengthy financial documents into a structured Retrieval-Augmented Generation (RAG) task. We construct a triangular consistency metric, assessing pairwise alignment between retrieved evidence, reasoning chains, and decisions to serve as a validity filter for noisy market returns. We explore two reward integration strategies: Fixed-effect Semantic Reward (FSR) for stable alignment signals, and Dynamic-effect Semantic Reward (DSR) for coupled magnitude optimization. Experiments on different country asset selection demonstrate that our paradigm reduces reward hacking, with DSR achieving superior cross-market generalization while maintaining the highest reasoning consistency.

Complexity vs Empirical Score

• Math Complexity: 6.0/10
• Empirical Rigor: 6.0/10 (capped from 8.0: transaction costs not modelled; see penalty flags)
• Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper employs advanced concepts from reinforcement learning (RL), large language models (LLMs), and structured verification metrics, indicating substantial mathematical sophistication. It also demonstrates strong empirical rigor through multi-market experiments (A-Share and US), backtest-style portfolio construction, and the proposal of specific, implementable frameworks like the Triangular Verification Protocol and semantic reward strategies.

Research Flowchart

  flowchart TD
  A["Research Goal:<br>Extend LLM RL to<br>Stochastic Financial Markets"] --> B["Input: Financial Docs &<br>Market Data"]
  B --> C{"Process: Trade-R1 Framework<br>via Process-Level Verification"}
  C --> D["Step 1: RAG-based<br>Evidence Retrieval"]
  D --> E["Step 2: Triangular Consistency<br>Verification Metric"]
  E --> F["Step 3: Reward Integration<br>FSR / DSR"]
  F --> G["Outcome: Trade-R1 Model<br>Reduced Reward Hacking<br>DSR: Superior Generalization"]

Cited References (50)

1. The Rise and Potential of Large Language Model Based Agents: A Survey (2023) — link
2. Gpt-4 technical report (2023) — link
3. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning (2025) — link
4. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning (2025) — link
5. Code llama: Open foundation models for code (2023) — link
6. Can chatgpt forecast stock price movements? return predictability and large language models — → 2304.07619
7. FinGPT: Open-Source Financial Large Language Models (2025) — link
8. Efficient capital markets: A review of theory and empirical work (1970)

6 of 50 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.