LLM Agents Play Pokemon Showdown

In a recent study on multi-agent collaboration, artificial intelligence agents were pitted against each other in a 2v2 format of the Pokémon Showdown game. The goal was to evaluate how well these agents could communicate, negotiate actions, and coordinate their strategies. The researchers wrapped the official Pokémon Showdown simulator into a Gym-style environment, allowing four players to control three Pokémon each - two on their team and one on the opposing side.

Each turn, an agent would read a plain-English summary of the battle and a numbered list of legal moves, then choose a move to execute. The agents could either communicate freely with their partner, send one message each, remain silent, or have a single model controlling both Pokémon. The results showed that when teammates were allowed to talk freely, they won 64% of their battles against a max-damage bot, compared to 42% when they couldn't communicate at all.

Even a single message each (52%) outperformed the no-talk scenario. While one model controlling both Pokémon performed slightly better (54%) than no talk, it was still behind the benefits of free communication. However, the study also found that communicating came at a cost - free talk used 978k prompt tokens per battle, three times the 259k used in no-talk battles.

The researchers noted that the environment could be used as a benchmark to compare different models and even train them to encourage collaborative behaviors through reinforcement learning.