Skip to content

Commit 860ebda

Browse files
markmandelmarkmandel
committed
Packed: Fleet scaled down removes nodes from least used Nodes
This implements the strategy such that when a Fleet is scaled down, and has the "Packed" strategy, it removed GameServers from Nodes that have the least GameServers running on them. This also fixes up some issues with the OpenAPI validation as well. THis looks like it's working. Go team.
1 parent cccdc8a commit 860ebda

13 files changed

Lines changed: 207 additions & 153 deletions

File tree

‎docs/scheduling_autoscaling.md‎

Lines changed: 35 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -7,14 +7,23 @@
77
Table of Contents
88
=================
99

10-
* [Fleet Autoscaling](#fleet-autoscaling)
11-
* [Autoscalng Concepts](#autoscalng-concepts)
12-
* [Allocation Scheduling](#allocation-scheduling)
13-
* [Fleet Scheduling](#fleet-scheduling)
14-
* [Packed](#packed)
15-
* [Allocation Scheduling Strategy](#allocation-scheduling-strategy)
16-
* [Distributed](#distributed)
17-
* [Allocation Scheduling Stategy](#allocation-scheduling-stategy)
10+
* [Scheduling and Autoscaling](#scheduling-and-autoscaling)
11+
* [Table of Contents](#table-of-contents)
12+
* [Fleet Autoscaling](#fleet-autoscaling)
13+
* [Autoscaling Concepts](#autoscaling-concepts)
14+
* [Allocation Scheduling](#allocation-scheduling)
15+
* [Pod Scheduling](#pod-scheduling)
16+
* [Fleet Scale Down Strategy](#fleet-scale-down-strategy)
17+
* [Fleet Scheduling](#fleet-scheduling)
18+
* [Packed](#packed)
19+
* [Allocation Scheduling Strategy](#allocation-scheduling-strategy)
20+
* [Pod Scheduling Strategy](#pod-scheduling-strategy)
21+
* [Fleet Scale Down Strategy](#fleet-scale-down-strategy-1)
22+
* [Distributed](#distributed)
23+
* [Allocation Scheduling Strategy](#allocation-scheduling-strategy-1)
24+
* [Pod Scheduling Strategy](#pod-scheduling-strategy-1)
25+
* [Fleet Scale Down Strategy](#fleet-scale-down-strategy-2)
26+
1827

1928
Scheduling and autoscaling go hand in hand, as where in the cluster `GameServers` are provisioned
2029
impacts how to autoscale fleets up and down (or if you would even want to)
@@ -42,6 +51,11 @@ Each `GameServer` is backed by a Kubernetes [`Pod`](https://kubernetes.io/docs/c
4251
refers to the strategy that is in place that determines which node in the Kubernetes cluster the Pod is assigned to,
4352
when it is created.
4453

54+
### Fleet Scale Down Strategy
55+
56+
Fleet Scale Down strategy refers to the order in which the `GameServers` that belong to a `Fleet` are deleted,
57+
when Fleets are shrunk in size.
58+
4559
## Fleet Scheduling
4660

4761
There are two scheduling strategies for Fleets - each designed for different types of Kubernetes Environments.
@@ -74,8 +88,7 @@ for the infrastructure you use.
7488
It attempts to _pack_ as much as possible into the smallest set of nodes, to make
7589
scaling infrastructure down as easy as possible.
7690
77-
Currently, Allocation scheduling is the only aspect this strategy affects, but in future releases it will
78-
also affect `GameServer` `Pod` scheduling, and `Fleet` scale down scheduling as well.
91+
This affects Allocation Scheduling, Pod Scheduling and Fleet Scale Down Scheduling.
7992
8093
#### Allocation Scheduling Strategy
8194
@@ -91,6 +104,11 @@ topology. This attempts to group together `GameServer` Pods within as few nodes
91104
> The default Kubernetes scheduler doesn't do a perfect job of packing, but it's a good enough job for what we need -
92105
at least at this stage.
93106

107+
#### Fleet Scale Down Strategy
108+
109+
With the "Packed" strategy, Fleets will remove `Ready` `GameServers` from Nodes with the _least_ number of `Ready` and
110+
`Allocated` `GameServers` on them. Attempting to empty Nodes so that they can be safely removed.
111+
94112
### Distributed
95113

96114
```yaml
@@ -118,8 +136,7 @@ on bare metal, and the cluster size rarely changes, if at all.
118136
This attempts to distribute the load across the entire cluster as much as possible, to take advantage of the static
119137
size of the cluster.
120138

121-
Currently, the only thing the scheduling strategy affects is Allocation scheduling, but in future releases it will
122-
also affect `GameServer` `Pod` scheduling, and `Fleet` scaledown scheduling as well.
139+
This affects Allocation Scheduling, Pod Scheduling and Fleet Scale Down Scheduling.
123140

124141
#### Allocation Scheduling Strategy
125142

@@ -129,4 +146,9 @@ number of allocated `GameServers` on them.
129146
#### Pod Scheduling Strategy
130147

131148
Under the "Distributed" strategy, `Pod` scheduling is provided by the default Kubernetes scheduler, which will attempt
132-
to distribute the `GameServer` `Pods` across as many nodes as possible.
149+
to distribute the `GameServer` `Pods` across as many nodes as possible.
150+
151+
#### Fleet Scale Down Strategy
152+
153+
With the "Distributed" strategy, Fleets will remove `Ready` `GameServers` from Nodes with at random, to ensure
154+
a distributed load is maintained.

‎examples/simple-udp/fleet.yaml‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,6 @@ spec:
2525
spec:
2626
ports:
2727
- name: default
28-
portPolicy: "dynamic"
2928
containerPort: 7654
3029
template:
3130
spec:

‎install/helm/agones/templates/crds/_gameserverspecvalidation.yaml‎

Lines changed: 4 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -57,7 +57,7 @@ properties:
5757
ports:
5858
title: array of ports to expose on the game server container
5959
type: array
60-
minItems: 0 # make this 1 in 0.4.0
60+
minItems: 1
6161
required:
6262
- containerPort
6363
items:
@@ -91,34 +91,11 @@ properties:
9191
type: integer
9292
minimum: 1
9393
maximum: 65535
94-
portPolicy: # remove this in 0.4.0
95-
title: the port policy that will be applied to the game server
96-
description: |
97-
portPolicy has two options:
98-
- "dynamic" (default) the system allocates a free hostPort for the gameserver, for game clients to connect to
99-
- "static", user defines the hostPort that the game client will connect to. Then onus is on the user to ensure that the
100-
port is available. When static is the policy specified, `hostPort` is required to be populated
94+
scheduling:
10195
type: string
10296
enum:
103-
- dynamic
104-
- static
105-
protocol: # remove this in 0.4.0
106-
title: Protocol being used. Defaults to UDP. TCP is the only other option
107-
type: string
108-
enum:
109-
- UDP
110-
- TCP
111-
containerPort: # remove this in 0.4.0
112-
title: The port that is being opened on the game server process
113-
type: integer
114-
minimum: 1
115-
maximum: 65535
116-
hostPort: # remove this in 0.4.0
117-
title: The port exposed on the host
118-
description: Only required when `portPolicy` is "static". Overwritten when portPolicy is "dynamic".
119-
type: integer
120-
minimum: 1
121-
maximum: 65535
97+
- Packed
98+
- Distributed
12299
health:
123100
type: object
124101
title: Health checking for the running game server

‎install/helm/agones/templates/crds/fleet.yaml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,9 +43,15 @@ spec:
4343
replicas:
4444
type: integer
4545
minimum: 0
46+
scheduling:
47+
type: string
48+
enum:
49+
- Packed
50+
- Distributed
4651
strategy:
4752
properties:
4853
type:
54+
type: string
4955
enum:
5056
- Recreate
5157
- RollingUpdate

‎install/helm/agones/templates/crds/fleetautoscaler.yaml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -50,8 +50,9 @@ spec:
5050
- type
5151
properties:
5252
type:
53+
type: string
5354
enum:
54-
- Buffer
55+
- Buffer
5556
buffer:
5657
required:
5758
- maxReplicas

‎install/helm/agones/templates/crds/gameserverset.yaml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,5 +44,10 @@ spec:
4444
replicas:
4545
type: integer
4646
minimum: 0
47+
scheduling:
48+
type: string
49+
enum:
50+
- Packed
51+
- Distributed
4752
template:
4853
{{- include "gameserver.validation" . | indent 14 }}

‎install/yaml/install.yaml‎

Lines changed: 25 additions & 82 deletions
Original file line numberDiff line numberDiff line change
@@ -184,9 +184,15 @@ spec:
184184
replicas:
185185
type: integer
186186
minimum: 0
187+
scheduling:
188+
type: string
189+
enum:
190+
- Packed
191+
- Distributed
187192
strategy:
188193
properties:
189194
type:
195+
type: string
190196
enum:
191197
- Recreate
192198
- RollingUpdate
@@ -234,7 +240,7 @@ spec:
234240
ports:
235241
title: array of ports to expose on the game server container
236242
type: array
237-
minItems: 0 # make this 1 in 0.4.0
243+
minItems: 1
238244
required:
239245
- containerPort
240246
items:
@@ -268,34 +274,11 @@ spec:
268274
type: integer
269275
minimum: 1
270276
maximum: 65535
271-
portPolicy: # remove this in 0.4.0
272-
title: the port policy that will be applied to the game server
273-
description: |
274-
portPolicy has two options:
275-
- "dynamic" (default) the system allocates a free hostPort for the gameserver, for game clients to connect to
276-
- "static", user defines the hostPort that the game client will connect to. Then onus is on the user to ensure that the
277-
port is available. When static is the policy specified, `hostPort` is required to be populated
277+
scheduling:
278278
type: string
279279
enum:
280-
- dynamic
281-
- static
282-
protocol: # remove this in 0.4.0
283-
title: Protocol being used. Defaults to UDP. TCP is the only other option
284-
type: string
285-
enum:
286-
- UDP
287-
- TCP
288-
containerPort: # remove this in 0.4.0
289-
title: The port that is being opened on the game server process
290-
type: integer
291-
minimum: 1
292-
maximum: 65535
293-
hostPort: # remove this in 0.4.0
294-
title: The port exposed on the host
295-
description: Only required when `portPolicy` is "static". Overwritten when portPolicy is "dynamic".
296-
type: integer
297-
minimum: 1
298-
maximum: 65535
280+
- Packed
281+
- Distributed
299282
health:
300283
type: object
301284
title: Health checking for the running game server
@@ -421,8 +404,9 @@ spec:
421404
- type
422405
properties:
423406
type:
407+
type: string
424408
enum:
425-
- Buffer
409+
- Buffer
426410
buffer:
427411
required:
428412
- maxReplicas
@@ -515,7 +499,7 @@ spec:
515499
ports:
516500
title: array of ports to expose on the game server container
517501
type: array
518-
minItems: 0 # make this 1 in 0.4.0
502+
minItems: 1
519503
required:
520504
- containerPort
521505
items:
@@ -549,34 +533,11 @@ spec:
549533
type: integer
550534
minimum: 1
551535
maximum: 65535
552-
portPolicy: # remove this in 0.4.0
553-
title: the port policy that will be applied to the game server
554-
description: |
555-
portPolicy has two options:
556-
- "dynamic" (default) the system allocates a free hostPort for the gameserver, for game clients to connect to
557-
- "static", user defines the hostPort that the game client will connect to. Then onus is on the user to ensure that the
558-
port is available. When static is the policy specified, `hostPort` is required to be populated
559-
type: string
560-
enum:
561-
- dynamic
562-
- static
563-
protocol: # remove this in 0.4.0
564-
title: Protocol being used. Defaults to UDP. TCP is the only other option
536+
scheduling:
565537
type: string
566538
enum:
567-
- UDP
568-
- TCP
569-
containerPort: # remove this in 0.4.0
570-
title: The port that is being opened on the game server process
571-
type: integer
572-
minimum: 1
573-
maximum: 65535
574-
hostPort: # remove this in 0.4.0
575-
title: The port exposed on the host
576-
description: Only required when `portPolicy` is "static". Overwritten when portPolicy is "dynamic".
577-
type: integer
578-
minimum: 1
579-
maximum: 65535
539+
- Packed
540+
- Distributed
580541
health:
581542
type: object
582543
title: Health checking for the running game server
@@ -647,6 +608,11 @@ spec:
647608
replicas:
648609
type: integer
649610
minimum: 0
611+
scheduling:
612+
type: string
613+
enum:
614+
- Packed
615+
- Distributed
650616
template:
651617
required:
652618
- spec
@@ -691,7 +657,7 @@ spec:
691657
ports:
692658
title: array of ports to expose on the game server container
693659
type: array
694-
minItems: 0 # make this 1 in 0.4.0
660+
minItems: 1
695661
required:
696662
- containerPort
697663
items:
@@ -725,34 +691,11 @@ spec:
725691
type: integer
726692
minimum: 1
727693
maximum: 65535
728-
portPolicy: # remove this in 0.4.0
729-
title: the port policy that will be applied to the game server
730-
description: |
731-
portPolicy has two options:
732-
- "dynamic" (default) the system allocates a free hostPort for the gameserver, for game clients to connect to
733-
- "static", user defines the hostPort that the game client will connect to. Then onus is on the user to ensure that the
734-
port is available. When static is the policy specified, `hostPort` is required to be populated
694+
scheduling:
735695
type: string
736696
enum:
737-
- dynamic
738-
- static
739-
protocol: # remove this in 0.4.0
740-
title: Protocol being used. Defaults to UDP. TCP is the only other option
741-
type: string
742-
enum:
743-
- UDP
744-
- TCP
745-
containerPort: # remove this in 0.4.0
746-
title: The port that is being opened on the game server process
747-
type: integer
748-
minimum: 1
749-
maximum: 65535
750-
hostPort: # remove this in 0.4.0
751-
title: The port exposed on the host
752-
description: Only required when `portPolicy` is "static". Overwritten when portPolicy is "dynamic".
753-
type: integer
754-
minimum: 1
755-
maximum: 65535
697+
- Packed
698+
- Distributed
756699
health:
757700
type: object
758701
title: Health checking for the running game server

‎pkg/apis/stable/v1alpha1/gameserver.go‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -118,7 +118,7 @@ type GameServerSpec struct {
118118
// Health configures health checking
119119
Health Health `json:"health,omitempty"`
120120
// Scheduling strategy. Defaults to "Packed".
121-
Scheduling SchedulingStrategy `json:"scheduling"`
121+
Scheduling SchedulingStrategy `json:"scheduling,omitempty"`
122122
// Template describes the Pod that will be created for the GameServer
123123
Template corev1.PodTemplateSpec `json:"template"`
124124
}

‎pkg/apis/stable/v1alpha1/gameserverset.go‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -57,7 +57,7 @@ type GameServerSetSpec struct {
5757
// Replicas are the number of GameServers that should be in this set
5858
Replicas int32 `json:"replicas"`
5959
// Scheduling strategy. Defaults to "Packed".
60-
Scheduling SchedulingStrategy `json:"scheduling"`
60+
Scheduling SchedulingStrategy `json:"scheduling,omitempty"`
6161
// Template the GameServer template to apply for this GameServerSet
6262
Template GameServerTemplateSpec `json:"template"`
6363
}

0 commit comments

Comments
 (0)