Set Termination Grace Period to write_timeout for functions to allow them to complete during a scale down event.

[alexellis]

Description

Set Termination Grace Period to write_timeout for functions to allow them to complete during a scale down event.

Motivation and Context

Prior to this change, a function could only drain for 30 seconds, then Kubernetes would forcibly kill it. After this change, in--flight function invocations can drain according to the write_timeout environment variable.

For example, this function would have failed to complete in-flight requests that ran for 1m30s.

functions:
  go-long:
    lang: golang-middleware
    handler: ./go-long
    image: alexellis2/graceful:0.2.1-newest
    environment:
      write_timeout: 2m
      read_timeout: 2m
      exec_timeout: 2m
      handler_wait_duration: 1m30s
      healthcheck_interval: 5s
    annotations:
      topic: "pipeline.subscription"
    labels:
      com.openfaas.scale.min: 1
      com.openfaas.scale.max: 1

Fixes #853 #637

How Has This Been Tested?

I deployed openfaas with global timeouts of 2m as per the "Extended timeout" tutorial in the docs.

Then I deployed the go-long function with a 1m wait and 2m max write_timeout/exec_timeout.

Then I used hey and an extended -t (timeout flag) to schedule 5 requests.

At this point I scale down the function with kubectl scale and monitored the logs.

Normally, the default of 30 seconds would have caused the functions to exit, but instead they stayed running and the pod remained in a terminating status.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I've read the CONTRIBUTION guide
• I have signed-off my commits with git commit -s
• I have added tests to cover my changes.
• All new and existing tests passed.

Also related to this change is a watchdog update to allow the Pod to exit early, if all in-flight requests have completed:

openfaas/of-watchdog@31e1d32

[alexellis]

To test this patch see: How I test OpenFaaS changes with Kubernetes

[LucasRoesler]

Just one question about how we structure the code, but it otherwise looks good. If you don't want to split it into a helper, just let me know and I will approve the PR

[alexellis]

@LucasRoesler I think you also suggested adding 500ms +/- to the default write timeout for jitter.

[alexellis]

@kevin-lindsay-1 would you like to take this for a spin?

You may need to delete your functions to try this.

[kevin-lindsay-1]

@alexellis works good for me.

My testing procedure used Tilt to avoid needing to push the image to a registry

For the sake of anyone interested, here's how I did it:

1. copy your existing deployment's yaml from k8s, save it to a file named gateway-dep.yaml
2. change the image for the faas-netes container to simply be faas-netes; tilt will figure it out
3. put the file in the faas-netes project root
4. create a Tiltfile with the following contents:

# copied a `gateway-dep.yaml` from an existing deployment
k8s_yaml('./gateway-dep.yaml')

# build the faas-netes image
docker_build('faas-netes', '.')

# hack to copy over the .git folder
# https://github.com/tilt-dev/tilt/issues/2169
local_resource(
    'hack-copy_git',
    cmd='cp -R .git .newgit'
)

# deploy the gateway in kubernetes, tilt detects the newly built image
k8s_resource(
    'gateway',
    resource_deps=[
        'hack-copy_git'
    ]
)

5. update the docker image to RUN mv .newgit .git after the COPY . .
6. tilt up
7. delete function(s), redeploy them to see changes

For anyone who might want to use this as an example, please note that tilt recently added a kubectl_build command which uses buildkit inside of k8s, so you don't have to duplicate images on your machine. Haven't used it yet, but probably will.

[alexellis]

@alexellis works good for me.

💪

Thank you Kevin