Paper: arXiv 2508.03910

Authors: Caio de Souza Barbosa Costa, Anna Helena Reali Costa

Abstract

Recently, reinforcement learning has achieved remarkable results in various domains, including robotics, games, natural language processing, and finance. In the financial domain, this approach has been applied to tasks such as portfolio optimization, where an agent continuously adjusts the allocation of assets within a financial portfolio to maximize profit. Numerous studies have introduced new simulation environments, neural network architectures, and training algorithms for this purpose. Among these, a domain-specific policy gradient algorithm has gained significant attention in the research community for being lightweight, fast, and for outperforming other approaches. However, recent studies have shown that this algorithm can yield inconsistent results and underperform, especially when the portfolio does not consist of cryptocurrencies. One possible explanation for this issue is that the commonly used state normalization method may cause the agent to lose critical information about the true value of the assets being traded. This paper explores this hypothesis by evaluating two of the most widely used normalization methods across three different markets (IBOVESPA, NYSE, and cryptocurrencies) and comparing them with the standard practice of normalizing data before training. The results indicate that, in this specific domain, the state normalization can indeed degrade the agent’s performance.

Complexity vs Empirical Score

  • Math Complexity: 5.0/10
  • Empirical Rigor: 6.0/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper presents moderate mathematical formulation in its reinforcement learning setup and portfolio equations, while demonstrating strong empirical rigor by testing multiple normalization methods across three distinct financial markets (IBOVESPA, NYSE, cryptocurrencies) and measuring performance impact.

Research Flowchart

  flowchart TD
  A["Research Goal: Evaluate<br>State Normalization Methods<br>for RL Portfolio Optimization"] --> B["Data Collection:<br>3 Markets (IBOVESPA, NYSE, Crypto)"]
  B --> C["Methodology: Compare<br>2 Normalization Methods<br>vs. Standard Pre-normalization"]
  C --> D["Computational Process:<br>Train RL Agent<br>using Policy Gradient Algorithm"]
  D --> E{"Evaluate Agent<br>Performance across Markets"}
  E --> F["Key Finding 1:<br>State Normalization Degrades<br>Agent Performance"]
  E --> G["Key Finding 2:<br>Standard Pre-normalization<br>is Superior in this Domain"]

Cited References (157)

  1. TensorFlow : A System for Large-Scale Machine Learning (2016)
  2. The Buy and Hold Investment Strategy : Building and Evaluating Passive Portfolios from Iraq Stock Exchange (2023) — link
  3. Continuous Control with Stacked Deep Dynamic Recurrent Reinforcement Learning for Portfolio Optimization (2020) — link
  4. Deep Reinforcement Learning and Mean-Variance Strategies for Responsible Portfolio Optimization (2024) — link
  5. Optuna: A Next-generation Hyperparameter Optimization Framework (2019) — link
  6. An Adaptive Portfolio Trading System: A Risk-Return Portfolio Optimization Using Recurrent Reinforcement Learning with Expected Maximum Drawdown (2017) — link
  7. Selimamrouni/ Deep-Portfolio-Management-Reinforcement-Learning : V2.0 (2022) — link
  8. Hidden-Layer Configurations in Reinforcement Learning Models for Stock Portfolio Optimization (2025) — link
and 149 more
  1. Ranaroussi/Quantstats (2023)
  2. Quality Attributes in Edge Computing for the Internet of Things : A Systematic Mapping Study (2021) — link
  3. Deep Reinforcement Learning for Portfolio Management of Markets with a Dynamic Number of Assets (2021) — link
  4. Deep Reinforcement Trading with Predictable Returns (2023) — link
  5. Fantastic MASs and Where to Find Them: First Results and Lesson Learned (2023)
  6. OpenAI Gym (2016) — link
  7. Algorithmic and High-Frequency Trading (2015)
  8. A Primer on Partially Observable Markov Decision Processes ( POMDPs ) (2021) — link
  9. Reinforcement Learning with Convolutional Reservoir Computing (2020) — link
  10. XGBoost : A Scalable Tree Boosting System (2016) — link
  11. Graph Time-series Modeling in Deep Learning : A Survey (2024) — link
  12. On the Properties of Neural Machine Translation : Encoder-Decoder Approaches (2014) — link
  13. Xception: Deep Learning with Depthwise Separable Convolutions (2017) — link
  14. A Study of Low-Volatility PortfolioConstruction Methods (2014) — link
  15. Intention Is Choice with Commitment (1990) — link
  16. Support-Vector Networks (1995) — link
  17. A Comparison of the Basic Principles and Behavioural Aspects of Akka , JaCaMo and Jade Development Frameworks (2018)
  18. POE : A General Portfolio Optimization Environment for FinRL (2023) — link
  19. RLPortfolio : Reinforcement Learning for~ Financial Portfolio Optimization (2025) — link
  20. Temporal fusion transformers para a precifica \c c ~a o de ativos financeiros (2022)
  21. Portfolio Constructions in Cryptocurrency Market: A CVaR-based Deep Reinforcement Learning Approach (2023) — link
  22. K- Nearest Neighbour Classifiers : 2nd Edition (with Python Examples) (2022) — link
  23. Deep Direct Reinforcement Learning for Financial Signal Representation and Trading (2017) — link
  24. Reinforcement Learning Applied to Trading Systems : A Survey (2022) — link
  25. Artificial Intelligence Applied to Stock Market Trading : A Review (2021) — link
  26. Limitations of Physics Informed Machine Learning for Nonlinear Two-Phase Transport in Porous Media (2020) — link
  27. Deep Echo State Network ( DeepESN ): A Brief Survey (2020) — link
  28. Continuous-Time Reinforcement Learning Approach for Portfolio Management with Time Penalization (2019) — link
  29. Bayesian Data Analysis (2015) — link
  30. A Survey on Software Agent Architectures (2013)
  31. Generative Adversarial Nets (2014)
  32. A Brief Review of Portfolio Optimization Techniques (2022) — link
  33. Gym-Mtsim (2021)
  34. Gym-Anytrading (2023)
  35. The Unreasonable Effectiveness of Data (2009) — link
  36. Recent Advances in Reinforcement Learning in Finance (2023) — link
  37. Dynamic Neural Networks : A Survey (2021) — link
  38. Stock Portfolio Management by Using Fuzzy Ensemble Deep Reinforcement Learning Algorithm (2023) — link
  39. Array Programming with NumPy (2020) — link
  40. Using Echo State Networks to Approximate Value Functions for Control (2021) — link
  41. Neural Networks : A Comprehensive Foundation (1999)
  42. Literature Review: Machine Learning Techniques Applied to Financial Market Prediction (2019) — link
  43. Rainbow: Combining Improvements in Deep Reinforcement Learning (2017) — link
  44. Distilling the Knowledge in a Neural Network (2015) — link
  45. Long Short-Term Memory (1997) — link
  46. Multilayer Feedforward Networks Are Universal Approximators (1989) — link
  47. Application of Evolutionary Computation for Rule Discovery in Stock Algorithmic Trading: A Literature Review (2015) — link
  48. Deep Reinforcement Learning for Stock Portfolio Optimization by Connecting with Modern Portfolio Theory (2023) — link
  49. A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem (2017) — link
  50. Deep Reinforcement Learning for Portfolio Selection (2024) — link
  51. ZhengyaoJiang / PGPortfolio (2024)
  52. A Mean-VaR Based Deep Reinforcement Learning Framework for Practical Algorithmic Trading (2023) — link
  53. An Asynchronous Advantage Actor-Critic Reinforcement Learning Method for Stock Selection and Portfolio Management (2018) — link
  54. Gradient Theory of Optimal Flight Paths (1960) — link
  55. Text Mining for Market Prediction: A Systematic Review (2014) — link
  56. Semi- Supervised Classification with Graph Convolutional Networks (2017) — link
  57. Global Market Portfolio 2023 (2023)
  58. The Stability , Scalability and Performance of Multi-agent Systems (1998) — link
  59. ChatGPT : A Meta-Analysis after 2.5 Months (2023) — link
  60. Investment Diversification as a Strategy for Reducing Investment Risk (2018) — link
  61. Deep Reinforcement Learning (2018) — link
  62. Understanding and Addressing Quality Attributes of Microservices Architecture: A Systematic Literature Review (2021) — link
  63. Online Portfolio Management via Deep Reinforcement Learning with High-Frequency Data (2023) — link
  64. A Deep Reinforcement Learning Framework For Financial Portfolio Management (2024) — link
  65. Adversarial Deep Reinforcement Learning in Portfolio Management (2018) — link
  66. Contrastive Learning and Reward Smoothing for Deep Portfolio Management (2023) — link
  67. Continuous Control with Deep Reinforcement Learning (2015) — link
  68. Dynamic Portfolio Rebalancing through Reinforcement Learning (2022) — link
  69. Multiagent-Based Deep Reinforcement Learning for Risk-Shifting Portfolio Management (2022) — link
  70. Practical Deep Reinforcement Learning Approach for Stock Trading (2018) — link
  71. ElegantRL-Podracer : Scalable and Elastic Library for Cloud-Native Deep Reinforcement Learning (2022) — link
  72. FinRL : Deep Reinforcement Learning Framework to Automate Trading in Quantitative Finance (2022) — link
  73. FinRL-Meta : Market Environments and Benchmarks for Data-Driven Financial Reinforcement Learning (2022) — link
  74. Multi-Agent Deep Reinforcement Learning Algorithm with Trend Consistency Regularization for Portfolio Management (2022) — link
  75. MACD Indicator Explained , with Formula , Examples , and Limitations
  76. MACD Indicator for Trading - Tutorial and Examples
  77. On the Maximum Drawdown of a Brownian Motion (2004) — link
  78. Systematic Review on Reinforcement Learning in the Field of Fintech — → 2305.07466
  79. Portfolio Selection (1952) — link
  80. Developing a Risk Management Approach Based on Reinforcement Training in the Formation of an Investment Portfolio (2023) — link
  81. Engineering Multi-Agent Systems : State of Affairs and the Road Ahead (2020) — link
  82. Using Deep Reinforcement Learning with Hierarchical Risk Parity for Portfolio Optimization (2023) — link
  83. Asynchronous Methods for Deep Reinforcement Learning (2016) — link
  84. Data Augmentation: A Comprehensive Survey of Modern Approaches (2022) — link
  85. Advancing Investment Frontiers : Industry-grade Deep Reinforcement Learning for Portfolio Optimization (2024) — link
  86. Does Reinforcement Learning Outperform Deep Learning and Traditional Portfolio Optimization Models in Frontier and Developed Financial Markets? (2023) — link
  87. MetaTrader : An Reinforcement Learning Approach Integrating Diverse Policies for Portfolio Optimization (2022) — link
  88. A Novel Multi-Source Information-Fusion Predictive Framework Based on Deep Neural Networks for Accuracy Enhancement in Stock Market Prediction (2021) — link
  89. Attention U-Net : Learning Where to Look for the Pancreas (2018) — link
  90. ChatGPT (2023)
  91. An Intelligent Financial Portfolio Trading Strategy Using Deep Q-learning (2020) — link
  92. Twin-System Recurrent Reinforcement Learning for Optimizing Portfolio Strategy (2024) — link
  93. PyTorch : An Imperative Style , High-Performance Deep Learning Library (2019) — link
  94. Are `` Market Neutral ’’ Hedge Funds Really Market Neutral ? (2009) — link
  95. Scikit-Learn: Machine Learning in Python (2018) — link
  96. Acceleration of Stochastic Approximation by Averaging (1992) — link
  97. Stable- Baselines3 : Reliable Reinforcement Learning Implementations (2021)
  98. A Modelling of Novel Approach for Stock Portfolio Management Using Unsupervised Algorithm and Deep Reinforcement Technique (2023) — link
  99. Relative Strength Index ( RSI ) Indicator Explained With Formula
  100. Normalized Online Learning (2013) — link
  101. Normalized Online Learning (2013) — link
  102. Introduction to Probability Models (2014) — link
  103. Curriculum Learning Empowered Reinforcement Learning for Graph-Based Portfolio Management: Performance Optimization and Comprehensive Analysis (2024) — link
  104. FINBERT-PT-BR: An 'a lise de sentimentos de textos em portugu ^e s referentes ao mercado financeiro (2022)
  105. Modeling Relational Data with Graph Convolutional Networks (2017) — link
  106. Trust Region Policy Optimization (2015) — link
  107. Proximal Policy Optimization Algorithms (2017) — link
  108. The Sharpe Ratio (1994) — link
  109. A Multi-Scale Temporal Feature Aggregation Convolutional Neural Network for Portfolio Management (2019) — link
  110. GPM : A Graph Convolutional Network Based Reinforcement Learning Framework for Portfolio Management (2022) — link
  111. Mastering the Game of Go without Human Knowledge (2017) — link
  112. Investigating the Impact of Data Normalization on Classification Performance (2020) — link
  113. Portfolio Optimization Using Reinforcement Learning : A Study of Implementation of Learning to Optimize (2023) — link
  114. Smart Toys Market Size & Share Analysis - Industry Research Report - Growth Trends
  115. Financial Portfolio Optimization with Online Deep Reinforcement Learning and Restricted Stacked Autoencoder— DeepBreath (2020) — link
  116. Deep Graph Convolutional Reinforcement Learning for Financial Portfolio Management – DeepPocket (2021) — link
  117. From Deterministic to Stochastic: An Interpretable Stochastic Model-Free Reinforcement Learning Framework for Portfolio Optimization (2023) — link
  118. Reinforcement Learning-Based Portfolio Optimization with Deterministic State Transition (2025) — link
  119. Combining Transformer Based Deep Reinforcement Learning with Black-Litterman Model for Portfolio Optimization — → 2402.16609
  120. GraphSAGE with Deep Reinforcement Learning for Financial Portfolio Optimization (2024) — link
  121. Reinforcement Learning : An Introduction (2018)
  122. Going Deeper with Convolutions (2015) — link
  123. Pandas-Dev/Pandas: Pandas (2023) — link
  124. Gymnasium (2023) — link
  125. Optimizing Automated Trading Systems with Deep Reinforcement Learning (2023) — link
  126. Some Mathematical Notes on Three-Mode Factor Analysis (1966) — link
  127. UCI Machine Learning Repository : Adult Data Set
  128. Deep Reinforcement Learning Methods for Portfolio Management : A Survey
  129. Moving Average Convergence-Divergence ( MACD ) Trading Rule : An Application in Nepalese Stock Market " NEPSE " (2020) — link
  130. Deep Reinforcement Learning with Double Q-learning (2015) — link
  131. Attention Is All You Need (2023) — link
  132. One-Hot-Encoding, o Que 'e ? (2018)
  133. Risk-Sensitive Policies for Portfolio Management (2022) — link
  134. Portfolio Trading System of Digital Currencies: A Deep Reinforcement Learning with Multidimensional Attention Gating Mechanism (2020) — link
  135. Tianshou: A Highly Modularized Deep Reinforcement Learning Library (2022) — link
  136. What Is EMA ?- Exponential Moving Average - Fidelity
  137. Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning (1992) — link
  138. Risk- Aware Reinforcement Learning for~ Multi-Period Portfolio Selection (2023) — link
  139. World Internet Users Statistics and 2023 World Population Stats
  140. A Comprehensive Survey on Graph Neural Networks (2021) — link
  141. Portfolio Management System in Equity Market Neutral Using Reinforcement Learning (2021) — link
  142. Relation- Aware Transformer for Portfolio Policy Learning (2020) — link
  143. Reinforcement Learning for POMDP Environments Using State Representation with Reservoir Computing (2022) — link
  144. Qlib: An AI-oriented Quantitative Investment Platform (2020) — link
  145. Deep Reinforcement Learning for Automated Stock Trading (2023)
  146. Deep Reinforcement Learning for Portfolio Management (2023) — link
  147. Model-Based Deep Reinforcement Learning for Dynamic Portfolio Optimization (2019) — link
  148. Financial Time Series Forecasting Using CNN and Transformer (2023) — link
  149. Imitation Learning : Progress , Taxonomies and Challenges (2022) — link

2 of 157 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.