Bayesian Robust Cooperative Multi-Agent Reinforcement Learning Against Unknown Adversaries
中文标题: 面向未知对手的贝叶斯鲁棒协同多智能体强化学习
英文摘要
This paper addresses robustness in cooperative multi-agent reinforcement learning (c-MARL) against deployment-time adversaries with unknown objectives. The authors propose a Bayesian Dec-POMDP game model with a continuum of adversarial types, each corresponding to a distinct attack objective. To make the problem tractable, they introduce a partitioning scheme that groups adversarial policies based on their performance against a reference c-MARL policy, reducing it to a finite-type Bayesian game. They develop a provably convergent externally constrained reinforcement learning algorithm to compute adversarial policies and use a simultaneous gradient update scheme to obtain robust Bayesian c-MARL policies. The resulting approach, BATPAL, is shown in experiments to outperform state-of-the-art baselines across diverse benchmarks and attack strategies.
中文摘要
该论文研究协同多智能体强化学习在部署时面对未知目标对手的鲁棒性问题。作者提出了一种具有连续对手类型的贝叶斯Dec-POMDP博弈模型，每种类型对应不同的攻击目标。为使其可求解，他们引入一种基于对抗策略相对于参考c-MARL策略性能的分区方案，将问题转化为有限类型贝叶斯博弈。他们开发了可证明收敛的外部约束强化学习算法来计算对抗策略，并采用同步梯度更新方案获得鲁棒贝叶斯c-MARL策略。实验表明，所得方法BATPAL在多种基准和攻击策略下均优于当前最先进的基线。
关键要点
Formulates robustness against unknown adversaries in c-MARL as a Bayesian Dec-POMDP game with a continuum of adversarial types.
将c-MARL中对未知对手的鲁棒性问题形式化为具有连续对手类型的贝叶斯Dec-POMDP博弈。
Proposes a partitioning scheme that reduces the continuum to finite types by clustering adversarial policies based on performance against a reference policy.
提出一种分区方案，根据对抗策略相对于参考策略的性能将其聚类，将连续类型简化为有限类型。
Introduces externally constrained reinforcement learning with convergence guarantees to compute adversarial policies.
引入具有收敛保证的外部约束强化学习来计算对抗策略。
Presents BATPAL, a simultaneous gradient update method for robust Bayesian c-MARL, which outperforms baselines in experiments.
提出BATPAL，一种用于鲁棒贝叶斯c-MARL的同步梯度更新方法，实验表现优于基线。