Multi-Agent Reinforcement Learning: AlphaZero Self Play Training
WorkshopOnline
Silicon Valley Generative AI ~ The AI Collective Network
Over the last three meetings (AlphaZero Introduction, MCTS Realtime Policy Improvement, AlphaZero Training Pipeline) we learned how to apply the Monte Carlo tree search algorithm defined in the following paper:
Danihelka, I., Guez, A., Schrittwieser, J., & Silver, D. (2022). Policy Improvement by Planning with Gumbel (ICLR 2022). https://openreview.net/forum?id=bERaNdoegnO
As a follow-up to the famous AlphaZero algorithm, this paper focuses on using search simulations to improve an existing policy/value function. We showed the details of how this improvement occurs and developed a training pipeline to use search simulations as a data collection tool and master an environment.
So far, we have used an MDP environment with a fixed opponent based on an extended tic-tac-toe game. This time, we turn our attention to the full two-player zero sum game scenario in which the search is used in conjunction with self-play to produce increasingly strong policies. We will show the modifications needed to the original search algorithm needed to handle player switching and then build a training pipeline based on self play.
In an MDP environment, measuring performance is straightforward and we can expect monotonic improvement over time. In the self-play setting, we have two competing agents and the overall success of one player may go through cycles. Therefore, we need to develop other criteria for determining training progress that involves studying the performance across a population of policies over time. We will begin to touch on section 9.9 of Multi-Agent Reinforcement Learning: Foundations and Modern Approaches which covers populations of agents learning simultaneously.
As usual you can find below links to the textbook, previous chapter notes, slides, and recordings of some of the previous meetings.
Meetup Links:
Recordings of Previous RL Meetings
Recordings of Previous MARL Meetings
Short RL Tutorials
My exercise solutions and chapter notes for Sutton-Barto
My MARL repos