[0.5.0-rc1] restarting allocation due to template change permanently fails

[jippi]

Nomad version

Nomad v0.5.0-rc1 ('a8c8199e413d387021a15d7a1400c8b8372124d6+CHANGES')

Operating system and Environment details

Distributor ID: Debian
Description: Debian GNU/Linux 7.9 (wheezy)
Release: 7.9
Codename: wheezy

Linux web02 4.7.6 #1 SMP Sun Oct 2 13:15:49 UTC 2016 x86_64 GNU/Linux

Issue

Changing a key in consul, triggering a consul-template change using template{} permanently crashes all allocations when trying to restart them

Error: failed to start task 'server' for alloc 'f967d730-614d-346b-c9a8-651bfd65ed97': Failed to re-create container server-f967d730-614d-346b-c9a8-651bfd65ed97; aborting

Stopping the job and creating it again works flawlessly.

Trying multiple times, it seem be be random if the task fails or not. Did the same change 5 times, and 3 times both allocations fails, 1 times one fail, and 1 time none failed and restarted correctly.

Smells like a race condition somewhere?

Reproduction steps

Create job, change consul kv to force a restart of the allocation due to config change

Nomad Server logs (if appropriate)

https://gist.github.com/jippi/b2760c3342c9cbce9e863fbc72ff7997#file-log-txt

Job file (if appropriate)

Job HCL: https://gist.github.com/jippi/b2760c3342c9cbce9e863fbc72ff7997#file-job-hcl
Alloc JSON: https://gist.github.com/jippi/b2760c3342c9cbce9e863fbc72ff7997#file-alloc-json

[dadgar]

Got it to reproduce. Looks like an issue with the docker driver.

[dadgar]

Thanks for the great bug report!

[jippi]

@dadgar great the issue could be reproduced so fast! It's really paying off to run everything in debug mode for you guys! :)

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.