GameServer stucks in Shutdown state preventing a rolling update to complete

[runtarm]

What happened:

I've been seeing a GameServer stucks in Shutdown state from time to time. The DeletionTimestamp was set. The underlying pod had already gone, but the finalizer still exists in the GameServer object which blocks GameServer deletion.

When this happens, it prevents a GameServerSet (that the GameServer belongs to) from scaling down, which in turn causing a rolling update of a fleet to not complete i.e. mixed versions of GameServer are available for allocation.

What you expected to happen:

The GameServer in Shutdown state should always be removed after its underlying pod is gone.

How to reproduce it (as minimally and precisely as possible):

We suspect this is caused by a race condition. No idea how to reproduce it deterministically yet.

Anything else we need to know?:

One time when it happened, I saw an event from this line emitted but the following log entry is not there.

Our theory: There is a race condition that a pod deletion event arrived before the podLister cache is updated, so it skipped removing the finalizer at this line.

Normally, the race condition shouldn't be a problem since there is a periodic resync to ensure states are eventually consistent. But the condition here may prevent the resync to go through.

A suggestion from @markmandel is to change the condition to something more akin to:

    if oldGs.Status.State != newGs.Status.State || !newGs.ObjectMeta.DeletionTimestamp.IsZero() {
        c.enqueueGameServerBasedOnState(newGs)
    }

Environment:

• Agones version: 1.16
• Kubernetes version (use kubectl version): 1.19
• Cloud provider or hardware configuration: GKE
• Install method (yaml/helm): yaml
• Troubleshooting guide log(s): (see above)
• Others:

[markmandel]

I think the e2e test TestAutoscalerStressCreate is being hit with this issue, which is why I think this test is being flaky (and what I think will be a good way to test that this issue is resolved).

[WVerlaek]

We just ran into this as well, had three GameServers stuck in a shutdown state since 2 April, Agones version 1.18. Pods were gone and deletion timestamps were set on the GS but blocked by the agones finalizers still being present. I think the suggested change looks sensible