无忧文档

Instance-based state identification for reinforcement learning

Instance-Based State Identi cation for Reinforcement Learning Department of Computer Science University of Rochester Rochester, NY 14627-0226 mccallum@cs.rochester.edu

R. Andrew McCallum

Abstract This paper presents instance-based state identi cation, an approach to reinforcement learning and hidden state that builds disambiguating amounts of short-term memory on-line, and also learns with an order of magnitude fewer training steps than several previous approaches. Inspired by a key similaritybetween learning with hidden state and learning in continuous geometrical spaces, this approach uses instance-based (or\memory-based") learning, a method that has worked well in continuous spaces.

1 BACKGROUND AND RELATED WORK When a robot's next course of action depends on information that is hidden from the sensors because of problems such as occlusion, restricted range, bounded eld of view and limited attention, the robot su ers from hidden state. More formally, we say a reinforcement learning agent su ers from the hidden state problem if the agent's state representation is non-Markovian with respect to actions and utility. The hidden state problem arises as a case of perceptual aliasing: the mapping between states of the world and sensations of the agent is not one-to-one Whitehead, 1992]. If the agent's perceptual system produces the same outputs for two world states in which di erent actions are required, and if the agent's state representation consists only of its percepts, then the agent will fail to choose correct actions. Note that even if an agent's state representation includes some internal state beyond its

相关文档
热门文档
评论