ByteGen: A Tokenizer-Free Generative Model for Orderbook Events in Byte Space

Paper: arXiv 2508.02247

Authors: Yang Li, Zhi Chen

Abstract

Generative modeling of high-frequency limit order book (LOB) dynamics is a critical yet unsolved challenge in quantitative finance, essential for robust market simulation and strategy backtesting. Existing approaches are often constrained by simplifying stochastic assumptions or, in the case of modern deep learning models like Transformers, rely on tokenization schemes that affect the high-precision, numerical nature of financial data through discretization and binning. To address these limitations, we introduce ByteGen, a novel generative model that operates directly on the raw byte streams of LOB events. Our approach treats the problem as an autoregressive next-byte prediction task, for which we design a compact and efficient 32-byte packed binary format to represent market messages without information loss. The core novelty of our work is the complete elimination of feature engineering and tokenization, enabling the model to learn market dynamics from its most fundamental representation. We achieve this by adapting the H-Net architecture, a hybrid Mamba-Transformer model that uses a dynamic chunking mechanism to discover the inherent structure of market messages without predefined rules. Our primary contributions are: 1) the first end-to-end, byte-level framework for LOB modeling; 2) an efficient packed data representation; and 3) a comprehensive evaluation on high-frequency data. Trained on over 34 million events from CME Bitcoin futures, ByteGen successfully reproduces key stylized facts of financial markets, generating realistic price distributions, heavy-tailed returns, and bursty event timing. Our findings demonstrate that learning directly from byte space is a promising and highly flexible paradigm for modeling complex financial systems, achieving competitive performance on standard market quality metrics without the biases of tokenization.

Complexity vs Empirical Score

• Math Complexity: 6.0/10
• Empirical Rigor: 8.0/10
• Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper introduces a novel, mathematically dense hybrid Mamba-Transformer architecture with dynamic chunking and operates on raw byte streams, but it is backed by strong empirical validation on 34 million real-world events with comprehensive backtest-ready metrics.

Research Flowchart

  flowchart TD
  A["Research Goal: Generative modeling of high-frequency LOB dynamics without tokenization or feature engineering."] --> B["Data Input: Raw byte streams of LOB events<br>(34M events from CME Bitcoin futures)"]
  B --> C["Methodology: ByteGen Framework<br>• Packed binary format (32 bytes)<br>• Autoregressive next-byte prediction<br>• H-Net architecture (Mamba-Transformer hybrid)"]
  C --> D["Computational Process: Dynamic chunking<br>Discovers inherent market structure without predefined rules."]
  D --> E["Key Findings: Competitive performance<br>• Realistic price distributions<br>• Heavy-tailed returns<br>• Bursty event timing<br>(Without tokenization biases)"]

Cited References (29)

1. LOB-Bench: Benchmarking Generative AI for Finance – an Application to Limit Order Book Data (2025) — link
2. MarketGPT: Developing a Pre-trained transformer (GPT) for Modeling Financial Time Series (2024) — link
3. Generative AI for End-to-End Limit Order Book Modelling: A Token-Level Autoregressive Generative Model of Message Flow Using a Deep State Space Network (2023) — link
4. Dynamic Chunking for End-to-End Hierarchical Sequence Modeling (2025) — link
5. Volatility clustering in financial markets: empirical facts and agent-based models (2007)
6. A stochastic model for order book dynamics (2010)
7. Algorithmic and high-frequency trading (2015)
8. Universal features of price formation in financial markets: perspectives from deep learning (2019)

4 of 29 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.