Optimistic Multi-Agent Policy Gradient

Wenshuai Zhao1, Yi Zhao1, Zhiyuan Li2, Juho Kannala1,3, Joni Pajarinen1,
1Aalto University, 2University of Electronic Science and Technology of China, 3University of Oulu
Paper arXiv Code

Abstract

Relative overgeneralization (RO) occurs in cooperative multi-agent learning tasks when agents converge towards a suboptimal joint policy due to overfitting to suboptimal behaviors of other agents. No methods have been proposed for addressing RO in multi-agent policy gradient (MAPG) methods although these methods produce state-of-the-art results. To address this gap, we propose a general, yet simple, framework to enable optimistic updates in MAPG methods that alleviate the RO problem. Our approach involves clipping the advantage to eliminate negative values, thereby facilitating optimistic updates in MAPG. The optimism prevents individual agents from quickly converging to a local optimum. Additionally, we provide a formal analysis to show that the proposed method retains optimality at a fixed point. In extensive evaluations on a diverse set of tasks including the Multi-agent MuJoCo and Overcooked benchmarks, our method outperforms strong baselines on 13 out of 19 tested tasks and matches the performance on the rest.

Experiments

Matrix Game

The matrix games present severe relative overgeneralization problem. The way our method solves the RO problem is well illustrated by the following figure.


Multi-Agent MuJoCo

Our method consistently outperforms the baselines on the complex MA-MuJoCo benchmark. Interesting, our method usually learns slower but eventually converges to a better policy.

Overcooked

On another benchmark Overcooked with discrete action spaces, our method also shows improved performance.

Why Optimistic DQN not working

Similar optimistic update has been used in DQN based methods. However, the optimism exacerbates the Q overestimation and thus fails showing the benefit of optimism. In contrast, optimistic MAPG methods avoid this problem naturally with the on-policy value estimation.

BibTeX

@inproceedings{zhao2024optimistic,
        title={Optimistic Multi-Agent Policy Gradient},
        author={Zhao, Wenshuai and Zhao, Yi and Li, Zhiyuan and Kannala, Juho and Pajarinen, Joni},
        booktitle={Proceedings of the International Conference on Machine Learning},
        year={2024}
      }