Environments
The map contains apples and lemons. The first player (red) is very sensitive and scores 5 for the team for an apple (green square) and −5 for a lemon (yellow square). The second (blue), less sensitive player scores 1 for the team for an apple and −1 for a lemon. There is a wall of lemons between the players and the apples. Apples and lemons disappear when collected, and the environment resets when all apples are eaten. It is important that the sensitive agent eats the apples while the less sensitive agent should leave them to its team mate but clear the way by eating obstructing lemons.
Reference Paper : Value-Decomposition Networks For Cooperative Multi-Agent Learning ( Section 4.2)
Action Space: 0: Down, 1: Left, 2: Up , 3: Right, 4: Noop
Agent Observation : Agent Coordinate + 3x3 mask around the agent + Steps in env.
Best Score: NA
Versions:
| Name | Description |
|---|---|
| Checkers-v0 | Each agent receives only it's local observation |
| Checkers-v1 | Each agent receives local observation all other agents |
It's a grid world environment having n agents where each agent wants to move their corresponding home location ( marked in boxes outlined in same colors).
The challenging part of the game is to pass through the narrow corridor through which only one agent can pass at a time. They need to coordinate to not block the pathway for the other. A reward of +5 is given to each agent for reaching their home cell.
The episode ends when both agent has reached their home state or for a maximum of 100 steps in environment.
Action Space: 0: Down, 1: Left, 2: Up , 3: Right, 4: Noop
Agent Observation : Agent Coordinate + Steps in env.
Best Score: NA
Reference Paper : Value-Decomposition Networks For Cooperative Multi-Agent Learning ( Section 4.2)
Versions:
| Name | Description |
|---|---|
| Switch2-v0 | Each agent receives only it's local position coordinates |
| Switch2-v1 | Each agent receives position coordinates of all other agents |
Similar variants are available for Switch4
| Switch2-v0 | Switch4-v0 |
|---|---|
 |
 |
Predator-prey involves a grid world, in which multiple predators attempt to capture randomly moving prey. Agents have a 5 × 5 view and select one of five actions ∈ {Left, Right, Up, Down, Stop} at each time step. Prey move according to selecting a uniformly random action at each time step. We define the “catching” of a prey as when the prey is within the cardinal direction of at least one predator. Each agent’s observation includes its own coordinates, agent ID, and the coordinates of the prey relative to itself, if observed. The agents can separate roles even if the parameters of the neural networks are shared by agent ID. We test with two different grid worlds: (i) a 5 × 5 grid world with two predators and one prey, and (ii) a 7 × 7 grid world with four predators and two prey. We modify the general predator-prey, such that a positive reward is given only if multiple predators catch a prey simultaneously, requiring a higher degree of cooperation. The predators get a team reward of 1 if two or more catch a prey at the same time, but they are given negative reward −P, if only one predator catches the prey as shown in Figure 7. We experimented with three varying P vales, where P = 0.5, 1.0, 1.5. The terminating condition of this task is when a prey is caught with more than one predator. The prey that has been caught is regenerated at random positions whenever the task terminates, and the game proceeds over fixed 100 steps.
Action Space: 0: Down, 1: Left, 2: Up , 3: Right, 4: Noop
Best Score: NA
| Name | Description |
|---|---|
| PredatorPrey5x5-v0 | |
| PredatorPrey5x5-v1 | |
| PredatorPrey5x5-v2 | |
| PredatorPrey5x5-v3 |
Similar variants are available for PredatorPrey7x7
| PredatorPrey5x5-v0 | PredatorPrey7x7-v0 |
|---|---|
 |
 |
We simulate a simple battle involving two opposing teams in a 15×15 grid as shown in Fig. 2(middle). Each team consists of m = 5 agents and their initial positions are sampled uniformly in a 5 × 5 square around the team center, which is picked uniformly in the grid. At each time step, an agent can perform one of the following actions: move one cell in one of four directions; attack another agent by specifying its ID j (there are m attack actions, each corresponding to one enemy agent); or do nothing. If agent A attacks agent B, then B’s health point will be reduced by 1, but only if B is inside the firing range of A (its surrounding 3 × 3 area). Agents need one time step of cooling down after an attack, during which they cannot attack. All agents start with 3 health points, and die when their health reaches 0. A team will win if all agents in the other team die. The simulation ends when one team wins, or neither of teams win within 40 time steps (a draw).
The model controls one team during training, and the other team consist of bots that follow a hardcoded policy. The bot policy is to attack the nearest enemy agent if it is within its firing range. If not, it approaches the nearest visible enemy agent within visual range. An agent is visible to all bots if it is inside the visual range of any individual bot. This shared vision gives an advantage to the bot team.
When input to a model, each agent is represented by a set of one-hot binary vectors {i, t, l, h, c} encoding its unique ID, team ID, location, health points and cooldown. A model controlling an agent also sees other agents in its visual range (3 × 3 surrounding area). The model gets reward of -1 if the team loses or draws at the end of the game. In addition, it also get reward of −0.1 times the total health points of the enemy team, which encourages it to attack enemy bots.
Reference : Learning Multiagent Communication with Backpropagation
Two Player Pong game.
Action Space: 0: Noop, 1: Up, 2: Down
Agent Observation : Agent Coordinate + Ball location ( head and tail)
Best Score: NA
The task tests if two agents can synchronize their behaviour, when picking up objects and returning them to a drop point. In the Fetch task both players start on the same side of the map and have pickup points on the opposite side. A player scores 3 points for the team for pick-up, and another 5 points for dropping off the item at the drop point near the starting position. Then the pickup is available to either player again. It is optimal for the agents to cycle such that when one player reaches the pickup point the other returns to base, to be ready to pick up again.
Best Score: NA
This consists of a 4-way junction on a 14 × 14 grid. At each time step, new cars enter the grid with probability p_arrive from each of the four directions. However, the total number of cars at any given time is limited to Nmax = 10. Each car occupies a single cell at any given time and is randomly assigned to one of three possible routes (keeping to the right-hand side of the road). At every time step, a car has two possible actions: gas which advances it by one cell on its route or brake to stay at its current location. A car will be removed once it reaches its destination at the edge of the grid.
Two cars collide if their locations overlap. A collision incurs a reward rcoll = −10, but does not affect the simulation in any other way. To discourage a traffic jam, each car gets reward of τrtime = −0.01τ at every time step, where τ is the number time steps passed since the car arrived. Therefore, the total reward at time t is .
Each car is represented by one-hot binary vector set {n, l, r}, that encodes its unique ID, current location and assigned route number respectively. Each agent controlling a car can only observe other cars in its vision range (a surrounding 3 × 3 neighborhood), but it can communicate to all other cars. The state vector s_j for each agent is thus a concatenation of all these vectors, having dimension 32 × |n| × |l| × |r|.
Reference : Learning Multiagent Communication with Backpropagation
Best Score: NA