To master the game Ms. Pac-Man, Microsoft researchers have created an artificial intelligence-based system that learned how to get the maximum score on the legendary video game Ms. Pac-Man, using a divide-and-conquer method that could have broad implications for teaching AI agents to do complex tasks that augment human capabilities.
The team from Maluuba, a Canadian deep learning startup acquired by Microsoft earlier this year, used a branch of AI called reinforcement learning to play the Atari 2600 version of Ms. Pac-Man perfectly. Using that method, the team achieved the maximum score possible of 999,990.
To get the high score, the team divided the large problem of mastering Ms. Pac-Man into small pieces, which they then distributed among AI agents. That's similar to some theories of how the brain works, and it could have broad implications for teaching AIs to do complex tasks with limited information.
The method, which the Maluuba team calls Hybrid Reward Architecture, used more than 150 agents, each of which worked in parallel with the other agents to master Ms. Pac-Man. For example, some agents got rewarded for successfully finding one specific pellet, while others were tasked with staying out of the way of ghosts.
Then, the researchers created a top agent who took suggestions from all the agents and used them to decide where to move Ms. Pac-Man. The top agent took into account how many agents advocated for going in a certain direction, but it also looked at the intensity with which they wanted to make that move. For example, if 100 agents wanted to go right because that was the best path to their pellet, but three wanted to go left because there was a deadly ghost to the right, it would give more weight to the ones who had noticed the ghost and go left.
Harm Van Seijen, a research manager with Maluuba who is the lead author of a new paper about the achievement, said the best results were achieved when each agent acted very egotistically - for example, focused only on the best way to get to its pellet - while the top agent decided how to use the information from each agent to make the best move for everyone.
Rahul Mehrotra, a program manager at Maluuba, said figuring out how to win these types of videogames is actually quite complex, because of the huge variety of situations you can encounter while playing the game.
With reinforcement learning, an agent gets positive or negative responses for each action it tries, and learns through trial and error to maximize the positive responses, or rewards.
An AI-based system that uses supervised learning would learn how to come up with a proper response in a conversation by feeding it examples of good and bad responses. A reinforcement learning system, on the other hand, would be expected to learn appropriate responses from only high-level feedback, such as a person saying she enjoyed the conversation - a much more difficult task.
AI experts believe reinforcement learning could be used to create AI agents that can make more decisions on their own, allowing them to do more complex work and freeing up people for even more high-value work.