Skip to main content

Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observability

Author(s): Ghosh, Dibya; Rahme, Jad; Kumar, Aviral; Adams, Ryan P.; Levine, Sergey; et al

Download
To refer to this page use: http://arks.princeton.edu/ark:/88435/pr11c1tg26
Abstract: Generalization is a central challenge for the deployment of reinforcement learning (RL) systems in the real world. In this paper, we show that the sequential structure of the RL problem necessitates new approaches to generalization beyond the well-studied techniques used in supervised learning. While supervised learning methods can generalize effectively without explicitly accounting for epistemic uncertainty, we show that, perhaps surprisingly, this is not the case in RL. We show that generalization to unseen test conditions from a limited number of training conditions induces implicit partial observability, effectively turning even fully-observed MDPs into POMDPs. Informed by this observation, we recast the problem of generalization in RL as solving the induced partially observed Markov decision process, which we call the epistemic POMDP. We demonstrate the failure modes of algorithms that do not appropriately handle this partial observability, and suggest a simple ensemble-based technique for approximately solving the partially observed problem. Empirically, we demonstrate that our simple algorithm derived from the epistemic POMDP achieves significant gains in generalization over current methods on the Procgen benchmark suite.
Publication Date: 1-Jan-2021
Citation: Ghosh, D, Rahme, J, Kumar, A, Zhang, A, Adams, RP, Levine, S. (2021). Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observability. Advances in Neural Information Processing Systems, 31 (25502 - 25515
ISSN: 1049-5258
Pages: 25502 - 25515
Type of Material: Journal Article
Journal/Proceeding Title: Advances in Neural Information Processing Systems
Version: Final published version. This is an open access article.



Items in OAR@Princeton are protected by copyright, with all rights reserved, unless otherwise indicated.