Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Packed: Fleet scaled down removes GameServers from least used Nodes #401

Merged
merged 1 commit into from
Oct 30, 2018
+207 −153
Merged

Packed: Fleet scaled down removes GameServers from least used Nodes #401

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 35 additions & 13 deletions docs/scheduling_autoscaling.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,14 +7,23 @@
Table of Contents
=================

* [Fleet Autoscaling](#fleet-autoscaling)
* [Autoscalng Concepts](#autoscalng-concepts)
* [Allocation Scheduling](#allocation-scheduling)
* [Fleet Scheduling](#fleet-scheduling)
* [Packed](#packed)
* [Allocation Scheduling Strategy](#allocation-scheduling-strategy)
* [Distributed](#distributed)
* [Allocation Scheduling Stategy](#allocation-scheduling-stategy)
* [Scheduling and Autoscaling](#scheduling-and-autoscaling)
* [Table of Contents](#table-of-contents)
* [Fleet Autoscaling](#fleet-autoscaling)
* [Autoscaling Concepts](#autoscaling-concepts)
* [Allocation Scheduling](#allocation-scheduling)
* [Pod Scheduling](#pod-scheduling)
* [Fleet Scale Down Strategy](#fleet-scale-down-strategy)
* [Fleet Scheduling](#fleet-scheduling)
* [Packed](#packed)
* [Allocation Scheduling Strategy](#allocation-scheduling-strategy)
* [Pod Scheduling Strategy](#pod-scheduling-strategy)
* [Fleet Scale Down Strategy](#fleet-scale-down-strategy-1)
* [Distributed](#distributed)
* [Allocation Scheduling Strategy](#allocation-scheduling-strategy-1)
* [Pod Scheduling Strategy](#pod-scheduling-strategy-1)
* [Fleet Scale Down Strategy](#fleet-scale-down-strategy-2)


Scheduling and autoscaling go hand in hand, as where in the cluster `GameServers` are provisioned
impacts how to autoscale fleets up and down (or if you would even want to)
Expand Down Expand Up @@ -42,6 +51,11 @@ Each `GameServer` is backed by a Kubernetes [`Pod`](https://kubernetes.io/docs/c
refers to the strategy that is in place that determines which node in the Kubernetes cluster the Pod is assigned to,
when it is created.

### Fleet Scale Down Strategy

Fleet Scale Down strategy refers to the order in which the `GameServers` that belong to a `Fleet` are deleted,
when Fleets are shrunk in size.

## Fleet Scheduling

There are two scheduling strategies for Fleets - each designed for different types of Kubernetes Environments.
Expand Down Expand Up @@ -74,8 +88,7 @@ for the infrastructure you use.
It attempts to _pack_ as much as possible into the smallest set of nodes, to make
scaling infrastructure down as easy as possible.

Currently, Allocation scheduling is the only aspect this strategy affects, but in future releases it will
also affect `GameServer` `Pod` scheduling, and `Fleet` scale down scheduling as well.
This affects Allocation Scheduling, Pod Scheduling and Fleet Scale Down Scheduling.

#### Allocation Scheduling Strategy

Expand All @@ -91,6 +104,11 @@ topology. This attempts to group together `GameServer` Pods within as few nodes
> The default Kubernetes scheduler doesn't do a perfect job of packing, but it's a good enough job for what we need -
at least at this stage.

#### Fleet Scale Down Strategy

With the "Packed" strategy, Fleets will remove `Ready` `GameServers` from Nodes with the _least_ number of `Ready` and
`Allocated` `GameServers` on them. Attempting to empty Nodes so that they can be safely removed.

### Distributed

```yaml
Expand Down Expand Up @@ -118,8 +136,7 @@ on bare metal, and the cluster size rarely changes, if at all.
This attempts to distribute the load across the entire cluster as much as possible, to take advantage of the static
size of the cluster.

Currently, the only thing the scheduling strategy affects is Allocation scheduling, but in future releases it will
also affect `GameServer` `Pod` scheduling, and `Fleet` scaledown scheduling as well.
This affects Allocation Scheduling, Pod Scheduling and Fleet Scale Down Scheduling.

#### Allocation Scheduling Strategy

Expand All @@ -129,4 +146,9 @@ number of allocated `GameServers` on them.
#### Pod Scheduling Strategy

Under the "Distributed" strategy, `Pod` scheduling is provided by the default Kubernetes scheduler, which will attempt
to distribute the `GameServer` `Pods` across as many nodes as possible.
to distribute the `GameServer` `Pods` across as many nodes as possible.

#### Fleet Scale Down Strategy

With the "Distributed" strategy, Fleets will remove `Ready` `GameServers` from Nodes with at random, to ensure
a distributed load is maintained.
1 change: 0 additions & 1 deletion examples/simple-udp/fleet.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,6 @@ spec:
spec:
ports:
- name: default
portPolicy: "dynamic"
containerPort: 7654
template:
spec:
Expand Down
31 changes: 4 additions & 27 deletions install/helm/agones/templates/crds/_gameserverspecvalidation.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ properties:
ports:
title: array of ports to expose on the game server container
type: array
minItems: 0 # make this 1 in 0.4.0
minItems: 1
required:
- containerPort
items:
Expand Down Expand Up @@ -91,34 +91,11 @@ properties:
type: integer
minimum: 1
maximum: 65535
portPolicy: # remove this in 0.4.0
title: the port policy that will be applied to the game server
description: |
portPolicy has two options:
- "dynamic" (default) the system allocates a free hostPort for the gameserver, for game clients to connect to
- "static", user defines the hostPort that the game client will connect to. Then onus is on the user to ensure that the
port is available. When static is the policy specified, `hostPort` is required to be populated
scheduling:
type: string
enum:
- dynamic
- static
protocol: # remove this in 0.4.0
title: Protocol being used. Defaults to UDP. TCP is the only other option
type: string
enum:
- UDP
- TCP
containerPort: # remove this in 0.4.0
title: The port that is being opened on the game server process
type: integer
minimum: 1
maximum: 65535
hostPort: # remove this in 0.4.0
title: The port exposed on the host
description: Only required when `portPolicy` is "static". Overwritten when portPolicy is "dynamic".
type: integer
minimum: 1
maximum: 65535
- Packed
- Distributed
health:
type: object
title: Health checking for the running game server
Expand Down
6 changes: 6 additions & 0 deletions install/helm/agones/templates/crds/fleet.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,9 +43,15 @@ spec:
replicas:
type: integer
minimum: 0
scheduling:
type: string
enum:
- Packed
- Distributed
strategy:
properties:
type:
type: string
enum:
- Recreate
- RollingUpdate
Expand Down
3 changes: 2 additions & 1 deletion install/helm/agones/templates/crds/fleetautoscaler.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,8 +50,9 @@ spec:
- type
properties:
type:
type: string
enum:
- Buffer
- Buffer
buffer:
required:
- maxReplicas
Expand Down
5 changes: 5 additions & 0 deletions install/helm/agones/templates/crds/gameserverset.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,5 +44,10 @@ spec:
replicas:
type: integer
minimum: 0
scheduling:
type: string
enum:
- Packed
- Distributed
template:
{{- include "gameserver.validation" . | indent 14 }}
107 changes: 25 additions & 82 deletions install/yaml/install.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -184,9 +184,15 @@ spec:
replicas:
type: integer
minimum: 0
scheduling:
type: string
enum:
- Packed
- Distributed
strategy:
properties:
type:
type: string
enum:
- Recreate
- RollingUpdate
Expand Down Expand Up @@ -234,7 +240,7 @@ spec:
ports:
title: array of ports to expose on the game server container
type: array
minItems: 0 # make this 1 in 0.4.0
minItems: 1
required:
- containerPort
items:
Expand Down Expand Up @@ -268,34 +274,11 @@ spec:
type: integer
minimum: 1
maximum: 65535
portPolicy: # remove this in 0.4.0
title: the port policy that will be applied to the game server
description: |
portPolicy has two options:
- "dynamic" (default) the system allocates a free hostPort for the gameserver, for game clients to connect to
- "static", user defines the hostPort that the game client will connect to. Then onus is on the user to ensure that the
port is available. When static is the policy specified, `hostPort` is required to be populated
scheduling:
type: string
enum:
- dynamic
- static
protocol: # remove this in 0.4.0
title: Protocol being used. Defaults to UDP. TCP is the only other option
type: string
enum:
- UDP
- TCP
containerPort: # remove this in 0.4.0
title: The port that is being opened on the game server process
type: integer
minimum: 1
maximum: 65535
hostPort: # remove this in 0.4.0
title: The port exposed on the host
description: Only required when `portPolicy` is "static". Overwritten when portPolicy is "dynamic".
type: integer
minimum: 1
maximum: 65535
- Packed
- Distributed
health:
type: object
title: Health checking for the running game server
Expand Down Expand Up @@ -421,8 +404,9 @@ spec:
- type
properties:
type:
type: string
enum:
- Buffer
- Buffer
buffer:
required:
- maxReplicas
Expand Down Expand Up @@ -515,7 +499,7 @@ spec:
ports:
title: array of ports to expose on the game server container
type: array
minItems: 0 # make this 1 in 0.4.0
minItems: 1
required:
- containerPort
items:
Expand Down Expand Up @@ -549,34 +533,11 @@ spec:
type: integer
minimum: 1
maximum: 65535
portPolicy: # remove this in 0.4.0
title: the port policy that will be applied to the game server
description: |
portPolicy has two options:
- "dynamic" (default) the system allocates a free hostPort for the gameserver, for game clients to connect to
- "static", user defines the hostPort that the game client will connect to. Then onus is on the user to ensure that the
port is available. When static is the policy specified, `hostPort` is required to be populated
type: string
enum:
- dynamic
- static
protocol: # remove this in 0.4.0
title: Protocol being used. Defaults to UDP. TCP is the only other option
scheduling:
type: string
enum:
- UDP
- TCP
containerPort: # remove this in 0.4.0
title: The port that is being opened on the game server process
type: integer
minimum: 1
maximum: 65535
hostPort: # remove this in 0.4.0
title: The port exposed on the host
description: Only required when `portPolicy` is "static". Overwritten when portPolicy is "dynamic".
type: integer
minimum: 1
maximum: 65535
- Packed
- Distributed
health:
type: object
title: Health checking for the running game server
Expand Down Expand Up @@ -647,6 +608,11 @@ spec:
replicas:
type: integer
minimum: 0
scheduling:
type: string
enum:
- Packed
- Distributed
template:
required:
- spec
Expand Down Expand Up @@ -691,7 +657,7 @@ spec:
ports:
title: array of ports to expose on the game server container
type: array
minItems: 0 # make this 1 in 0.4.0
minItems: 1
required:
- containerPort
items:
Expand Down Expand Up @@ -725,34 +691,11 @@ spec:
type: integer
minimum: 1
maximum: 65535
portPolicy: # remove this in 0.4.0
title: the port policy that will be applied to the game server
description: |
portPolicy has two options:
- "dynamic" (default) the system allocates a free hostPort for the gameserver, for game clients to connect to
- "static", user defines the hostPort that the game client will connect to. Then onus is on the user to ensure that the
port is available. When static is the policy specified, `hostPort` is required to be populated
scheduling:
type: string
enum:
- dynamic
- static
protocol: # remove this in 0.4.0
title: Protocol being used. Defaults to UDP. TCP is the only other option
type: string
enum:
- UDP
- TCP
containerPort: # remove this in 0.4.0
title: The port that is being opened on the game server process
type: integer
minimum: 1
maximum: 65535
hostPort: # remove this in 0.4.0
title: The port exposed on the host
description: Only required when `portPolicy` is "static". Overwritten when portPolicy is "dynamic".
type: integer
minimum: 1
maximum: 65535
- Packed
- Distributed
health:
type: object
title: Health checking for the running game server
Expand Down
2 changes: 1 addition & 1 deletion pkg/apis/stable/v1alpha1/gameserver.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,7 @@ type GameServerSpec struct {
// Health configures health checking
Health Health `json:"health,omitempty"`
// Scheduling strategy. Defaults to "Packed".
Scheduling SchedulingStrategy `json:"scheduling"`
Scheduling SchedulingStrategy `json:"scheduling,omitempty"`
// Template describes the Pod that will be created for the GameServer
Template corev1.PodTemplateSpec `json:"template"`
}
Expand Down
2 changes: 1 addition & 1 deletion pkg/apis/stable/v1alpha1/gameserverset.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ type GameServerSetSpec struct {
// Replicas are the number of GameServers that should be in this set
Replicas int32 `json:"replicas"`
// Scheduling strategy. Defaults to "Packed".
Scheduling SchedulingStrategy `json:"scheduling"`
Scheduling SchedulingStrategy `json:"scheduling,omitempty"`
// Template the GameServer template to apply for this GameServerSet
Template GameServerTemplateSpec `json:"template"`
}
Expand Down
Loading