Paper: arXiv 2405.15975

Abstract

This paper introduces a novel stochastic control framework to enhance the capabilities of automated investment managers, or robo-advisors, by accurately inferring clients’ investment preferences from past activities. Our approach leverages a continuous-time model that incorporates utility functions and a generic discounting scheme of a time-varying rate, tailored to each client’s risk tolerance, valuation of daily consumption, and significant life goals. We address the resulting time inconsistency issue through state augmentation and the establishment of the dynamic programming principle and the verification theorem. Additionally, we provide sufficient conditions for the identifiability of client investment preferences. To complement our theoretical developments, we propose a learning algorithm based on maximum likelihood estimation within a discrete-time Markov Decision Process framework, augmented with entropy regularization. We prove that the log-likelihood function is locally concave, facilitating the fast convergence of our proposed algorithm. Practical effectiveness and efficiency are showcased through two numerical examples, including Merton’s problem and an investment problem with unhedgeable risks. Our proposed framework not only advances financial technology by improving personalized investment advice but also contributes broadly to other fields such as healthcare, economics, and artificial intelligence, where understanding individual preferences is crucial.

Complexity vs Empirical Score

  • Math Complexity: 9.0/10
  • Empirical Rigor: 3.0/10
  • Quadrant: Lab Rats — theoretically deep, empirically untested

Why this score: The paper employs advanced stochastic control, dynamic programming principles, and verification theorems in continuous time, resulting in high mathematical complexity. However, it lacks real-world backtesting, datasets, or performance metrics, relying instead on theoretical proofs and two numerical examples (Merton’s problem and unhedgeable risks), indicating low empirical rigor.

Research Flowchart

  flowchart TD
  A["Research Goal: Infer Investment Preferences<br>of Robo-Advisor Clients"] --> B["Methodology: Stochastic Control Framework"]
  
  B --> C["Inputs: Past Client Activity<br>Utility Functions & Discounting Schemes"]
  
  C --> D["Core Process: State Augmentation<br>Dynamic Programming Principle"]
  D --> E["Computational Process: MLE with Entropy Regularization"]
  
  E --> F{"Outcomes"}
  F --> G["Identifiability of Preferences"]
  F --> H["Concave Log-Likelihood & Fast Convergence"]
  F --> I["Personalized Investment Advice"]

Cited References (132)

  1. Consumption Theory in Terms of Revealed Preference (1948)
  2. Recovery of Preferences from Observed Wealth in a Single Realization (1997)
  3. Revealed Preference Theory (1966)
  4. Investment and Valuation Under Backward and Forward Dynamic Exponential Utilities in a Stochastic Factor Model (2007) — link
  5. An Exact Connection between Two Solvable SDEs and a Nonlinear Utility Stochastic PDE (2013) — link
  6. When Is a Linear Control System Optimal? (1964)
  7. Patient preferences for shared decisions: a systematic review (2012)
  8. Patient registries: utility, validity and inference (2010)
and 124 more
  1. Targeted advertising and consumer inference (2021)
  2. Consumer inferences and consumer preferences. The status of cognition and consciousness in consumer behavior theory (1985)
  3. Linear matrix inequalities in system and control theory (1994)
  4. Learning agents for uncertain environments (1998)
  5. Decisions with Multiple Objectives: Preferences and Value Trade-Offs (1976)
  6. Estimation of dynamic labor demand schedules under rational expectations (1978)
  7. Apprenticeship learning via inverse reinforcement learning (2004)
  8. Relative entropy inverse reinforcement learning (2011)
  9. Modeling Purposeful Adaptive Behavior with the Principle of Maximum Causal Entropy (2010)
  10. Nonlinear inverse reinforcement learning with gaussian processes (2011)
  11. Guided cost learning: Deep inverse optimal control via policy optimization (2016)
  12. Identifiability in inverse reinforcement learning (2021)
  13. Repeated Inverse Reinforcement Learning (2017)
  14. Towards resolving unidentifiability in inverse reinforcement learning (2016) — link
  15. Learning Robust Rewards with Adverserial Inverse Reinforcement Learning (2018)
  16. Reward Identification in Inverse Reinforcement Learning (2021) — link
  17. Generative adversarial imitation learning (2016)
  18. Identifiability and generalizability in constrained inverse reinforcement learning (2023)
  19. Towards Generalized Inverse Reinforcement Learning (2024) — link
  20. Myopia and Inconsistency in Dynamic Utility Maximization (1955)
  21. Consistent Planning1 (1968) — link
  22. A general theory of Markovian time inconsistent stochastic control problems (2010)
  23. Investments and forward utilities (2006)
  24. Behavioral portfolio selection in continuous time (2008)
  25. A Continuous Time Framework for Sequential Goal-Based Wealth Management (2024)
  26. A theory of Markovian time-inconsistent stochastic control in discrete time (2014)
  27. Deep reinforcement learning from human preferences (2017)
  28. On time-inconsistent stochastic control in continuous time (2017)
  29. The golden rule when preferences are time inconsistent (2010)
  30. Time-inconsistent stochastic linear–quadratic control (2012)
  31. Time-inconsistent stochastic linear-quadratic control: Characterization and uniqueness of equilibrium (2017)
  32. Time-inconsistent optimal control problems and the equilibrium HJB equation (2012) — link
  33. Me, myself and I: a general theory of non-Markovian time-inconsistent stochastic control for sophisticated agents (2023)
  34. Dynamic approaches for some time-inconsistent optimization problems (2017) — link
  35. Reinforcement learning with deep energy-based policies (2017)
  36. Optimal control of diffustion processes and hamilton-jacobi-bellman equations part I: the dynamic programming principle and application (1983) — link
  37. Optimal stochastic control of diffusion type processes and Hamilton-Jacobi-Bellman equations (1982)
  38. Maximum-likelihood inverse reinforcement learning with finite-time guarantees (2022)
  39. A solution approach to valuation with unhedgeable risks (2001)
  40. Optimal consumption in a growth model with the Cobb–Douglas production function (2009)
  41. A solvable two-dimensional degenerate singular stochastic control problem with nonconvex costs (2019)
  42. Controlled Markov processes and viscosity solutions (2006)
  43. Who benefits from robo-advising? Evidence from machine learning (2020)
  44. Robo-advising (2021)
  45. Bi-revealed utilities in a defaultable universe: A new point of view on consumption (2024)
  46. Recover Dynamic Utility from Monotonic Characteristic/Extremal Processes. (2019)
  47. Construction of an aggregate consistent utility, without Pareto optimality. Application to long-term yield curve modeling (2019)
  48. Recover dynamic utility from observable process: Application to the economic equilibrium (2021)
  49. Learning equilibrium mean-variance strategy (2023)
  50. Personalized robo-advising: Enhancing investment through client interaction (2022)
  51. Learning risk preferences from investment portfolios using inverse optimization (2023)
  52. Robo-advising: Learning investors’ risk preferences via portfolio choices (2021)
  53. Robo-advising: Enhancing investment with inverse optimization and deep reinforcement learning (2021)
  54. Maximum entropy inverse reinforcement learning. (2008)
  55. The promises and pitfalls of robo-advising (2019)
  56. Sqil: Imitation learning via reinforcement learning with sparse rewards (2019) — link
  57. f-irl: Inverse reinforcement learning via state marginal matching (2021)
  58. More risk-sensitive Markov decision processes (2014)
  59. UTILITY THEORY FRONT TO BACK—INFERRING UTILITY FROM AGENTS’CHOICES (2014)
  60. Adaptive discretization in online reinforcement learning (2023)
  61. On the theory of policy gradient methods: Optimality, approximation, and distribution shift (2021)
  62. The expected utility model: Its variants, purposes, evidence and limitations (1982)
  63. Conditionally elicitable dynamic risk measures for deep reinforcement learning (2022) — link
  64. A reinforcement learning based model for adaptive service quality management in e-commerce websites (2020)
  65. Impression Allocation for Combating Fraud in E-commerce Via Deep Reinforcement Learning with Action Norm Penalty. (2018)
  66. Reinforcement learning to rank in e-commerce search engine: Formalization, analysis, and application (2018)
  67. E-commerce recommendation with weighted expected utility (2020)
  68. Dynamic Pricing on E-commerce Platform with Deep Reinforcement Learning: A Field Experiment (2019) — link
  69. Minimax regret bounds for reinforcement learning (2017)
  70. A convex analytic approach to risk-aware Markov decision processes (2015)
  71. A theoretical analysis of deep Q-learning (2020)
  72. Sample efficient deep reinforcement learning via local planning (2023) — link
  73. Deep reinforcement learning with robust and smooth policy (2020)
  74. On the convergence and optimality of policy gradient for Markov coherent risk (2021) — link
  75. Reinforcement learning with dynamic convex risk measures (2023)
  76. Recent advances in reinforcement learning in finance (2023)
  77. Reinforcement learning and deep learning based lateral control for autonomous driving [application notes] (2019)
  78. Reinforcement learning in feature space: Matrix bandit, kernels, and regret bound (2020)
  79. Is pessimism provably efficient for offline rl? (2021)
  80. Mastering the game of Go with deep neural networks and tree search (2016)
  81. Playing atari with deep reinforcement learning (2013) — link
  82. Continuous control with deep reinforcement learning (2015) — link
  83. End-to-end training of deep visuomotor policies (2016)
  84. A dynamic analytic method for risk-aware controlled martingale problems (2023)
  85. Is Q-learning provably efficient? (2018)
  86. Hedging in incomplete markets with HARA utility (1997)
  87. Risk-sensitive Markov decision processes (1972)
  88. Provably efficient reinforcement learning with linear function approximation (2020)
  89. Sample-optimal parametric q-learning using linearly additive features (2019)
  90. Reinforcement learning with general value function approximation: Provably efficient approach via bounded eluder dimension (2020)
  91. Risk-sensitive reinforcement learning: Near-optimal risk-sample tradeoff in regret (2020)
  92. Regret Bounds for Risk-Sensitive Reinforcement Learning (2022)
  93. A sensitivity formula for risk-sensitive cost and the actor–critic algorithm (2001)
  94. Q-learning for risk-sensitive control (2002)
  95. Continuous-time stochastic control and optimization with financial applications (2009)
  96. Markov decision processes: discrete stochastic dynamic programming (2014)
  97. Risk-sensitive reinforcement learning (2014)
  98. Discounted approximations for risk-sensitive average criteria in Markov decision chains with finite state space (2011)
  99. Risk sensitive control of Markov processes in countable state space (1996)
  100. Stochastic approximation for risk-aware Markov decision processes (2020)
  101. Risk-sensitive reinforcement learning (2002)
  102. Robustness and risk-sensitivity in Markov decision processes (2012)
  103. On terminating Markov decision processes with a risk-averse objective function (2001)
  104. Risk-sensitive Markov control processes (2013)
  105. Risk-sensitive optimal control of quantum systems (2004)
  106. Exponential bellman equation and improved regret bounds for risk-sensitive reinforcement learning (2021)
  107. Near-Minimax-Optimal Risk-Sensitive Reinforcement Learning with CVaR (2023) — link
  108. Markov decision processes with average-value-at-risk criteria (2011)
  109. Risk-averse dynamic programming for Markov decision processes (2010)
  110. Decision problems with expected utility criteria, II: Stationarity (1977)
  111. Modern actuarial risk theory: using R (2008)
  112. The foundations of decision analysis (1968)
  113. Mean-variance analysis in portfolio choice and capital markets (2000)
  114. Near-optimal time and sample complexities for solving Markov decision processes with a generative model (2018)
  115. Model-based reinforcement learning with a generative model is minimax optimal (2020)
  116. Bayesian Inverse Reinforcement Learning. (2007)
  117. Efficient exploration of reward functions in inverse reinforcement learning via Bayesian optimization (2020)
  118. Efficient Bayesian Inverse Reinforcement Learning via Conditional Kernel Density Estimation (2022)
  119. Algorithms for inverse reinforcement learning. (2000)
  120. A survey of inverse reinforcement learning: Challenges, methods and progress (2021)
  121. A connection between generative adversarial networks, inverse reinforcement learning, and energy-based models (2016) — link
  122. Infinite time horizon maximum causal entropy inverse reinforcement learning (2014)
  123. Maximum entropy deep inverse reinforcement learning (2015) — link
  124. Iq-learn: Inverse soft-q learning for imitation (2021)

0 of 132 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.