Coordinating Team Rewards in OpenAI Five's Dota 2 Agents

OpenAI Five defeated the world champions of Dota 2 by solving a massive challenge in reinforcement learning: coordinating five autonomous agents over long time horizons without direct communication channels. This article explains the specific mechanisms behind this achievement, detailing how the system used a shared architecture, a specialized hyperparameter called "Team Spirit" to blend individual and collective rewards, and generalized advantage estimation to bridge the gap between immediate actions and a final victory.

In Dota 2, games routinely last 45 minutes, requiring thousands of sequential decisions where the ultimate reward—destroying the enemy's Ancient—occurs only at the very end. Instead of building a explicit communication protocol between the five agents, OpenAI trained a single neural network policy replicated five times. Each agent acts autonomously based on its own observations, which include the global state visible to the team. Coordination emerges naturally because all five agents optimize the exact same mathematical objective.

To teach the agents both individual skill and team coordination, OpenAI designed a reward function based on two levels of metrics:

• Individual Rewards: Collecting gold, gaining experience, getting last hits on creeps, and avoiding death.
• Team Rewards: Destroying enemy towers, securing the Roshan boss objective, winning team fights, and destroying the enemy Ancient.

If agents only optimize individual rewards, they play selfishly and lose the game. If they only optimize team rewards, the credit assignment problem makes it difficult for an individual agent to know if its specific actions contributed to the outcome.

To solve this, OpenAI introduced a hyperparameter called Team Spirit (\(\tau\)), ranging from 0 to 1. The actual reward each agent receives at any timestep is a mathematical blend of its own individual reward and the average reward of the entire five-agent team. The formula used is:

\[\text{Final Reward}_i = (1 - \tau) \cdot \text{Individual Reward}_i + \tau \cdot \text{Average Team Reward}\]

Early in the training process, Team Spirit was set low (near 0). This forced each agent to focus on mechanical micro-skills, such as farming gold and surviving in their specific lane, without worrying about complex team strategies. As the agents mastered basic mechanics, OpenAI annealed (gradually increased) the Team Spirit parameter toward 1.0.

At a Team Spirit of 1.0, an agent cares equally about its own gold and its teammates' gold. This mathematical shift caused the agents to spontaneously develop sacrificial and cooperative behaviors. Support heroes began sacrificing their own lives to save higher-value "carry" heroes, and agents willingly gave up gold and experience in lanes so that a teammate could become stronger for late-game fights.

Coordinating across a 45-minute time horizon required scaling Proximal Policy Optimization (PPO) with Generalized Advantage Estimation (GAE). Dota 2 runs at 30 frames per second, and OpenAI Five made decisions every fourth frame, resulting in roughly 20,000 steps per game.

To handle this long horizon, each agent utilized a single-layer, 4,096-unit Long Short-Term Memory (LSTM) network. The LSTM allowed agents to maintain a memory of the game state over time. Combined with a very high discount factor (\(\gamma\)), the value function could predict the probability of winning the game thousands of steps into the future.

By combining long-horizon value estimation with the Team Spirit reward structure, the five autonomous networks did not need to "talk" to each other. Every agent could look at the game map, process the state through an identical LSTM structure, and independently arrive at the same conclusion about what the team needed to do next to maximize their shared future reward.