Benchmarking Robustness and Generalization in Multi-Agent Systems: A Case Study on Neural MMO

Yangkun Chen; Chenghui Yu; Hengman Zhu; Shuai Liu; Yibing Zhang; Joseph Suarez; Liang Zhao; Jinke He; Jiaxin Chen; null More Authors

Benchmarking Robustness and Generalization in Multi-Agent Systems: A Case Study on Neural MMO

Yangkun Chen, Chenghui Yu, Hengman Zhu, Shuai Liu, Yibing Zhang, Joseph Suarez, Liang Zhao, Jinke He, Jiaxin Chen, More Authors

Interactive Intelligence

Research output: Contribution to journal › Conference article › Scientific › peer-review

72 Downloads (Pure)

Abstract

We present the results of the second Neural MMO challenge, hosted at IJCAI 2022, which received 1600+ submissions. This competition targets robustness and generalization in multi-agent systems: participants train teams of agents to complete a multi-task objective against opponents not seen during training. We summarize the competition design and results and suggest that, considering our work as a case study, competitions are an effective approach to solving hard problems and establishing a solid benchmark for algorithms. We will open-source our benchmark including the environment wrapper, baselines, a visualization tool, and selected policies for further research.

Original language	English
Pages (from-to)	2490-2492
Number of pages	3
Journal	Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS
Volume	2023-May
Publication status	Published - 2023
Event	22nd International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2023 - London, United Kingdom Duration: 29 May 2023 → 2 Jun 2023

Bibliographical note

Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care
Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.

Keywords

Benchmark
Competition
Multi-agent Reinforcement Learning

Access to Document

3545946.3598978Final published version, 3.33 MB

Cite this

@article{311c668f8cb742018105c19ab87aa888,

title = "Benchmarking Robustness and Generalization in Multi-Agent Systems: A Case Study on Neural MMO",

abstract = "We present the results of the second Neural MMO challenge, hosted at IJCAI 2022, which received 1600+ submissions. This competition targets robustness and generalization in multi-agent systems: participants train teams of agents to complete a multi-task objective against opponents not seen during training. We summarize the competition design and results and suggest that, considering our work as a case study, competitions are an effective approach to solving hard problems and establishing a solid benchmark for algorithms. We will open-source our benchmark including the environment wrapper, baselines, a visualization tool, and selected policies for further research.",

keywords = "Benchmark, Competition, Multi-agent Reinforcement Learning",

author = "Yangkun Chen and Chenghui Yu and Hengman Zhu and Shuai Liu and Yibing Zhang and Joseph Suarez and Liang Zhao and Jinke He and Jiaxin Chen and {More Authors}",

note = "Green Open Access added to TU Delft Institutional Repository {\textquoteleft}You share, we take care!{\textquoteright} – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public. ; 22nd International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2023 ; Conference date: 29-05-2023 Through 02-06-2023",

year = "2023",

language = "English",

volume = "2023-May",

pages = "2490--2492",

journal = "Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS",

issn = "1548-8403",

publisher = "International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS)",

}

Benchmarking Robustness and Generalization in Multi-Agent Systems: A Case Study on Neural MMO. / Chen, Yangkun; Yu, Chenghui; Zhu, Hengman et al.
In: Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS, Vol. 2023-May, 2023, p. 2490-2492.

Research output: Contribution to journal › Conference article › Scientific › peer-review

TY - JOUR

T1 - Benchmarking Robustness and Generalization in Multi-Agent Systems

T2 - 22nd International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2023

AU - Chen, Yangkun

AU - Yu, Chenghui

AU - Zhu, Hengman

AU - Liu, Shuai

AU - Zhang, Yibing

AU - Suarez, Joseph

AU - Zhao, Liang

AU - He, Jinke

AU - Chen, Jiaxin

AU - More Authors, null

N1 - Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.

PY - 2023

Y1 - 2023

N2 - We present the results of the second Neural MMO challenge, hosted at IJCAI 2022, which received 1600+ submissions. This competition targets robustness and generalization in multi-agent systems: participants train teams of agents to complete a multi-task objective against opponents not seen during training. We summarize the competition design and results and suggest that, considering our work as a case study, competitions are an effective approach to solving hard problems and establishing a solid benchmark for algorithms. We will open-source our benchmark including the environment wrapper, baselines, a visualization tool, and selected policies for further research.

AB - We present the results of the second Neural MMO challenge, hosted at IJCAI 2022, which received 1600+ submissions. This competition targets robustness and generalization in multi-agent systems: participants train teams of agents to complete a multi-task objective against opponents not seen during training. We summarize the competition design and results and suggest that, considering our work as a case study, competitions are an effective approach to solving hard problems and establishing a solid benchmark for algorithms. We will open-source our benchmark including the environment wrapper, baselines, a visualization tool, and selected policies for further research.

KW - Benchmark

KW - Competition

KW - Multi-agent Reinforcement Learning

UR - http://www.scopus.com/inward/record.url?scp=85171258676&partnerID=8YFLogxK

M3 - Conference article

AN - SCOPUS:85171258676

SN - 1548-8403

VL - 2023-May

SP - 2490

EP - 2492

JO - Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS

JF - Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS

Y2 - 29 May 2023 through 2 June 2023

ER -

Benchmarking Robustness and Generalization in Multi-Agent Systems: A Case Study on Neural MMO

Abstract

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Cite this