Modification of running allocs flaps consul health checks

[pearkes]

Nomad version

Nomad v0.3.1

Operating system and Environment details

Linux 3.13.0-52-generic #86-Ubuntu SMP x86_64 GNU/Linux

Issue

Nomad appears to be causing a flap to consul health checks when modifying a running alloc every time a job is submitted, even when the job file is unchanged. The flap of consul health checks causes availability issues. I would expect the service to not be registered/deregistered if the job file is the same.

Reproduction steps

1. Submit the job. You then have these running allocs:

nomad status service-example
ID          =  service-example
Name        =  service-example
Type        = service
Priority    = 50
Datacenters = dc1
Status      = running
Periodic    = false

==> Evaluations
ID        Priority  Triggered By  Status
78d9e40f  50        job-register  complete

==> Allocations
ID        Eval ID   Node ID   Task Group          Desired  Status
5ba22e50  78d9e40f  295a6717   service-example-green  run      running
88c082b0  78d9e40f  94305342   service-example-blue   run      running
b1f0a42a  78d9e40f  fe64412a   service-example-blue   run      running

Now, after resubmitting the unchanged jobs I get output like the following:

$ nomad run service-example.job
==> Monitoring evaluation "31f3538c"
    Evaluation triggered by job "slug-extract"
    Allocation "5ba22e50" modified: node "295a6717", group "service-example-blue"
    Allocation "88c082b0" modified: node "94305342", group "service-example-blue"
    Allocation "b1f0a42a" modified: node "fe64412a", group "service-example-green"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "31f3538c" finished with status "complete"

I can do this any number of times and it registers/deregisters every time, again, without changes to the job file.

Job file if appropriate

https://gist.github.com/pearkes/c218e7972c03965a99e0

[pearkes]

I update both the issue and description to better match the underlying issue which is deregistration/registration of Consul services causing a flap.

From a UX perspective, the "modification" of an unchanged job threw me off – if I didn't change anything, why is it modifying it? Shouldn't it say something like "processing, but no changes?". This led me to believe something else was going on than actually was. I understand it may be modifying it internally for good reasons but I just wonder if we can expose this in a clearer way that is less surprising.

[diptanu]

Fixed via #986

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.