🦋Self-Supervised Reversible Reinforcement Learning: A New Approach from Google AI
Reinforcement learning (RL) is great at solving problems from scratch, but it is not easy to train an agent to understand the reversibility of his actions. For example, robots should avoid activities that could damage them. To evaluate the reversibility of an action, one needs practical knowledge and understanding of the physics of the environment in which the RL agent exists. Therefore, Google AI researchers at the NeurIPS 2021 conference present a new way to approximate the reversibility of the actions of RL agents. This approach adds a separate reversibility assessment component to self-directed reinforcement learning from untagged data collected by agents. The model can be trained online (with the RL agent) or offline (from the interaction dataset) to guide RL policies towards reversible behavior. This can significantly improve the performance of RL agents when performing multiple tasks.
The reversibility component added to the RL procedure is extracted from interactions and is a model that can be trained separately from the agent itself. The model is trained on its own and does not require data markup indicating the reversibility of actions: the model itself learns which types of actions tend to be reversible from the context of the training data. This takes into account the probability of occurrence of events and priority as a proxy measure of the true reversibility, which can be learned from the dataset of interactions, even without rewarding the RL agent.
This method allows RL agents to predict reversibility of action by learning to simulate the temporal order of randomly selected trajectory events, resulting in better exploration and control. The method is self-checking, i.e. does not require prior knowledge of reversibility, which is suitable for different environments.
https://ai.googleblog.com/2021/11/self-supervised-reversibility-aware.html