Many people in the AI community focus on achieving flashy results, like building an agent that can win at Go or Jeopardy. This kind of work is impressive in terms of complexity. But it’s easy to forget another important axis of intelligence: generalization, the ability to handle a variety of tasks or operate in a range of situations. In 2020, I hope to see progress on building models that generalize.
My work involves using reinforcement learning to train robots that reason about how their actions will affect their environment. For example, I’d like to train a robot to perform a variety of tasks with a variety of objects, such as packing items into a box or sweeping trash into a dustpan. This can be hard to accomplish using RL.
In supervised learning, training an image recognizer on ImageNet’s 14 million pictures tends to result in a certain degree of generalization. In reinforcement learning, a model learns by interacting with a virtual environment and collecting data as it goes. To build the level of general skill we’re accustomed to seeing in models trained on ImageNet, we need to collect an ImageNet-size dataset for each new model. That’s not practical.
If we want systems trained by reinforcement learning to generalize, we need to design agents that can learn from offline datasets, not unlike ImageNet, as they explore an environment. And we need these pre-existing datasets to grow over time to reflect changes in the world, just as ImageNet has grown from its original 1 million images.
This is starting to happen. For example, robots can figure out how to use new objects as tools by learning from a dataset of their own interactions plus demonstrations performed by humans guiding a robot’s arm. We’re figuring out how to take advantage of data from other institutions. For instance, we collected a dataset of robots interacting with objects from seven different robot platforms across four institutions.
It’s exciting to see critical mass developing around generalization in reinforcement learning. If we can master these challenges, our robots will be a step closer to behaving intelligently in the real world, rather than doing intelligent-looking things in the lab.
Chelsea Finn is an assistant professor of computer science and electrical engineering at Stanford.