unhealthy deployment with poststart lifecycle

[tcdev0]

Nomad version

Nomad v1.0.3 (08741d9)

Operating system and Environment details

Ubuntu 18.04.5 LTS

Issue

Deployement with poststart task is immediately marked as unhealthy and still running as an active deployment until deadline hits.
The Service is registered healthy in Consul and reachable via traefik.

Removing the poststart job and the deployment is running fine.

I guess the poststart job is exiting too fast (under 10s) und therefore hits the min_healthy_time of 10s(default).

Reproduction steps

Job file

job "nginx" {

  datacenters = ["prod"]
  namespace   = "default"
  type        = "service"

  group "nginx" {
    count = 1
    service {
      name = "${BASE}"
      tags = [
        "traefik.enable=true",
        "traefik.http.routers.nginx.entrypoints=websecure",
        "traefik.http.routers.nginx.rule=Host(`nginx.domain`)",
        "traefik.http.routers.nginx.tls=true"
      ]
      port = "http"
      check {
        name     = "alive"
        type     = "tcp"
        interval = "2s"
        timeout  = "2s"
      }
    }

    network {
      port "http" {
        to = 80
      }
    }

    update {
      min_healthy_time  = "10s"
      healthy_deadline  = "5m"
      progress_deadline = "10m"
    }

    task "nginx-web" {
      driver = "docker"
      config {
        image = "nginx:latest"
        ports = ["http"]
      }
    }

    task "nginx-poststart" {
      driver = "docker"
      config {
        image   = "alpine:latest"
        command = "sh"
        args    = ["-c", "sleep 5 && echo 'running....' && exit 0;"]

      }
      lifecycle {
        hook    = "poststart"
        sidecar = false
      }
    }
  }
}

Enforcing the poststart job to wait for the min_healthy_time from the web service and exiting after, fixed the issue.
So i set the sleep timer slightly above the min_healthy_time = "10s" and it works.

    task "nginx-poststart" {
      driver = "docker"
      config {
        image   = "alpine:latest"
        command = "sh"
        args    = ["-c", "sleep 12 && echo 'running....' && exit 0;"]

      }
      lifecycle {
        hook    = "poststart"
        sidecar = false
      }
    }

Seen in 9361

Is it a bug or working as expected ?

Thanks.

[tgross]

HI @tcdev0. Looks like the fix for #9361 shipped in Nomad 1.0.2. This problem is slightly different, and looks like a bug emerging from the interaction of three things:

• A post start task
• That's not a sidecar (and so should be expected to exit)
• With a minimum healthy time in the update block

To make this work without your workaround (which looks sensible), we'd have to update the allocation health status to verify that the exit code is successful for post-stop tasks if we find they aren't "healthy" by the minimum healthy time.

[tommyalatalo]

We also observed this behavior today, having a task with a post start hook (not sidecar) and a min_healthy_time set in the update stanza. Will be following this, definitiely in need of a fix.

[maxless]

+1
Moreover, it feels weird that stuff in the update stanza is used to control prestart/poststart/poststop tasks. One of their intended uses as is shown in the documentation for lifecycle stanza is to just send HTTP requests to chat channels and exit. Which is precisely what it fails to do in the current implementation, marking allocation as unhealthy. I guess either there needs to be some flag in lifecycle, like "ignore_update", or separate task update block that overrides the group update block. It does work for prestart/poststop cases, so it's 2/3 job done.

[Vadim-Che]

Have Nomad v1.1.0 but faced the same issue. My jobs have Poststart and Poststop tasks. Poststop task is marked with "Task not running by deadline":

But the strangest thing is that there are other services, that are deployed by the same scheme and those are Ok (are not marked as failed):

Will this be fixed in recent future?

[Oloremo]

Hit that too on Nomad 1.1.5

[havenith]

+1
I have the same problem. The above sample fails on Nomad 1.1.6. And in our case we have a long running Java process and a short running(sub ten second), non-sidecar poststart task.
The deployment always fails with 'Task not running for min_healthy_time of 10s by deadline' even though the long running Java task is healthy in both Nomad and Consul.
Padding out the length of the poststart task doesn't seem to help though, so we don't have a workaround currently.
Is there a rough estimate on when a fix might come out?

[beautifulentropy]

Experiencing this too. As a new Nomad user I burned a ton of cycles on this before realizing this was just a bug.

[axsuul]

Still getting this issue on Nomad 1.2.0

[SunSparc]

Same. Nomad 1.2.0. I changed my poststart task to use the exec driver and handed it a bash script so I could use the sleep kludge that the OP used. This also got me past the problem, for now.

[ksklareski]

FYI issue persists in 1.2.4. Sleeping workaround still works.

[beautifulentropy]

A note for others who attempt to fix this with the a sleep of >=min_healthy_time. This workaround will result in a successful initial deployment. However, any subsequent update deployment where only new allocations are created forces create (not forces create/destroy update) will result in all previously healthy nodes becoming unhealthy; new allocations will be healthy though. The only real workaround I could find that truly avoids the issue is just making your existing post-start ephemeral hooks long-running sidecars instead.

OS macOS 12.1
Nomad v1.2.4
Consul v1.11.2

Repro Steps

1. Run the following job and wait for the first deployment to succeed.
2. Ensure the post-start allocation bar shows as having exited successfully.
3. Change count = 1 to count = 2
4. Run the updated job.
5. The original 'Healthy' allocations will now be 'Unhealthy' and the newly added allocations will be 'Healthy'.

job "foobar" {

  datacenters = ["dev"]
  type        = "service"

  group "foobar" {
    count = 1

    task "foo" {
      driver = "raw_exec"
      config {
        command = "sh"
        args    = ["-c", "sleep 5000000"]
      }
    }

    task "bar" {
      driver = "raw_exec"
      config {
        command = "sh"
        args    = ["-c", "sleep 10 && echo 'running....' && exit 0;"]
      }
      lifecycle {
        hook    = "poststart"
        sidecar = false
      }
    }
  }
}

Screenshots

(deployment 1)

(deployment 2)

[edudobay]

Should the same changes be applied to prestart tasks? The PR above only changed the behaviour for poststart tasks, but it seems like prestart should have been included as well.

[tgross]

@edudobay the code touched by #11945 already had a special case for prestart, and there wasn't any other discussion of prestart in this issue. If you're encountering something similar there, can you open a new issue for it? Thanks!

[shantanugadgil]

Is there a v1.2.7 anticipated? or should we just wait for v1.3.0 GA ?

[tgross]

We'll ship backports to 1.2.7 and 1.1.x when we ship 1.3.0 GA, which is happening soon!

[shantanugadgil]

Thanks! 🚀

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.