Multi-Agent Reinforcement Learning: Theory, Algorithms, and Future Directions.

Abstract

Reinforcement learning (RL) has been the driver behind many of the most significant advances in artificial intelligence over the past decade---ranging from achieving superhuman performance in complex games like Go and starcraft to applications in autonomous driving, robotics, and economic simulations. RL is even playing a crucial role in the fine tuning of large language models and the training of AI agents more broadly. Many of these problems, however, are fundamentally multi-agent in nature: an agent's success is inextricably linked to the decisions of others. Despite these empirical successes and a wealth of research on "single-agent" RL and its variants, multi-agent reinforcement learning (MARL) remains relatively under-explored theoretically with the presence of multiple learning agents giving rise to a unique set of challenges for algorithm design and analysis. 

This tutorial will give an overview of the research landscape in MARL, aiming to highlight the core theoretical principles that enable agents to learn and adapt in the presence of others. Using the formal framework of Markov games and building on a foundation in game theory, we will explore the different solution concepts and algorithms in the field. The discussion will explore the inherent inefficiencies of classical algorithms like fictitious play and policy gradient algorithms and build toward the principles underpinning modern, provably efficient learning methods for large games (i.e., algorithms that make use of function approximation like deep neural networks).  Ultimately, we will identify key open problems and promising new  research directions for the future of multi-agent learning.