keeping image latest always restarts service

[duritong]

Describe the Bug

#468 tried to fix an issue with noop but now introduced that the exec is always triggered if using latest.

Expected Behavior

Exec does not propagate a refresh if nothing updates, and nothing gets changed during a run:

Service gets notified by image - idea is that service should get restarted when image changes.

Docker::Image<| title == split($real_image,':')[0] |> ~> Service["docker-${name}"]

BUT service even gets restarted when image is not changed (Status: Image is up to date for registry.code.immerda.ch/immerda/container-images/gitlab:latest), because the notify resource still triggers as a change. Which has also the sideeffect that the node always counts as changed and never has 0 changes.

Notice: Check if image registry.code.immerda.ch/immerda/container-images/gitlab:latest is in-sync
Notice: /Stage[main]/Ib_gitlab::Base/Docker::Image[registry.code.immerda.ch/immerda/container-images/gitlab]/Notify[Check if image registry.code.immerda.ch/immerda/container-images/gitlab:latest is in-sync]/message: defined 'message' as 'Check if image registry.code.immerda.ch/immerda/container-images/gitlab:latest is in-sync'
Info: /Stage[main]/Ib_gitlab::Base/Docker::Image[registry.code.immerda.ch/immerda/container-images/gitlab]/Notify[Check if image registry.code.immerda.ch/immerda/container-images/gitlab:latest is in-sync]: Scheduling refresh of Exec[echo 'Update of registry.code.immerda.ch/immerda/container-images/gitlab:latest complete']
Debug: /Stage[main]/Ib_gitlab::Base/Docker::Image[registry.code.immerda.ch/immerda/container-images/gitlab]/Notify[Check if image registry.code.immerda.ch/immerda/container-images/gitlab:latest is in-sync]: The container Docker::Image[registry.code.immerda.ch/immerda/container-images/gitlab] will propagate my refresh event
Debug: /Stage[main]/Ib_gitlab::Base/Docker::Image[registry.code.immerda.ch/immerda/container-images/gitlab]/Exec[echo 'Update of registry.code.immerda.ch/immerda/container-images/gitlab:latest complete']: 'echo 'Update of registry.code.immerda.ch/immerda/container-images/gitlab:latest complete'' won't be executed because of failed check 'refreshonly'
Debug: Exec[echo 'Update of registry.code.immerda.ch/immerda/container-images/gitlab:latest complete'](provider=posix): Executing check '/usr/local/bin/update_docker_image.sh registry.code.immerda.ch/immerda/container-images/gitlab:latest'
Debug: Executing: '/usr/local/bin/update_docker_image.sh registry.code.immerda.ch/immerda/container-images/gitlab:latest'
Debug: /Stage[main]/Ib_gitlab::Base/Docker::Image[registry.code.immerda.ch/immerda/container-images/gitlab]/Exec[echo 'Update of registry.code.immerda.ch/immerda/container-images/gitlab:latest complete']/onlyif: Trying to pull repository registry.code.immerda.ch/immerda/container-images/gitlab ... 
Debug: /Stage[main]/Ib_gitlab::Base/Docker::Image[registry.code.immerda.ch/immerda/container-images/gitlab]/Exec[echo 'Update of registry.code.immerda.ch/immerda/container-images/gitlab:latest complete']/onlyif: sha256:f09e666210135cd89377767dad8cf06d5b4dc64cfbdd67c8f7ae3fc0f0b58ca9: Pulling from registry.code.immerda.ch/immerda/container-images/gitlab
Debug: /Stage[main]/Ib_gitlab::Base/Docker::Image[registry.code.immerda.ch/immerda/container-images/gitlab]/Exec[echo 'Update of registry.code.immerda.ch/immerda/container-images/gitlab:latest complete']/onlyif: Digest: sha256:f09e666210135cd89377767dad8cf06d5b4dc64cfbdd67c8f7ae3fc0f0b58ca9
Debug: /Stage[main]/Ib_gitlab::Base/Docker::Image[registry.code.immerda.ch/immerda/container-images/gitlab]/Exec[echo 'Update of registry.code.immerda.ch/immerda/container-images/gitlab:latest complete']/onlyif: Status: Image is up to date for registry.code.immerda.ch/immerda/container-images/gitlab:latest
Debug: /Stage[main]/Ib_gitlab::Base/Docker::Image[registry.code.immerda.ch/immerda/container-images/gitlab]/Exec[echo 'Update of registry.code.immerda.ch/immerda/container-images/gitlab:latest complete']/onlyif: No updates to registry.code.immerda.ch/immerda/container-images/gitlab:latest available. Currently on sha256:c075f1d3906617823dd2e805f9e42531e36959458ab1de2d5ce7e06b8f6271f1.
Debug: /Stage[main]/Ib_gitlab::Base/Docker::Image[registry.code.immerda.ch/immerda/container-images/gitlab]/Exec[echo 'Update of registry.code.immerda.ch/immerda/container-images/gitlab:latest complete']: 'echo 'Update of registry.code.immerda.ch/immerda/container-images/gitlab:latest complete'' won't be executed because of failed check 'onlyif'
Notice: /Stage[main]/Ib_gitlab::Base/Docker::Image[registry.code.immerda.ch/immerda/container-images/gitlab]/Exec[echo 'Update of registry.code.immerda.ch/immerda/container-images/gitlab:latest complete']: Triggered 'refresh' from 1 event
Debug: /Stage[main]/Ib_gitlab::Base/Docker::Image[registry.code.immerda.ch/immerda/container-images/gitlab]/Exec[echo 'Update of registry.code.immerda.ch/immerda/container-images/gitlab:latest complete']: The container Docker::Image[registry.code.immerda.ch/immerda/container-images/gitlab] will propagate my refresh event
Debug: Docker::Image[registry.code.immerda.ch/immerda/container-images/gitlab]: The container Class[Ib_gitlab::Base] will propagate my refresh event
Info: Docker::Image[registry.code.immerda.ch/immerda/container-images/gitlab]: Scheduling refresh of Service[docker-gitlab]

This is on latest master using puppet 6.9.0

CC @electrofelix

[electrofelix]

@duritong this reads as though the notify to the next resource is still being sent even though the onlyif test has failed and prevented the current resource from being updated.

I don't recall this happening in puppet 5, has something changed? If this was the case every puppet for our local dev-envs run would have restarted every container on each apply, which is definitely not what used to happen.

[duritong]

I recall something like that in the changelog for Puppet 6, but maybe I am remembering that wrong.

But yes, this is basically what is happening here.

[s10]

The behaviour of the module is the same with Puppet 5. Same issue was reported five months ago: #479

[den-is]

I encounter same issue
Really frustrating issue

# puppet --version
5.5.8

[Vincevrp]

I confirm this behavior on puppet 5.5.10.

[electrofelix]

I suspect the underlying issue is that having the image resource trigger the service restart on update is probably too inflexible and instead there should be a specific resource that is part of the image class that should be notified and instead of:

Docker::Image<| title == split($real_image,':')[0] |> ~> Service["docker-${name}"]

Use something like:

image_arg = split($real_image,':')[0]
Exec<| title == "echo 'Update of ${image_arg} complete'" |> ~> Service["docker-${name}"]

Ideally it would be nicer if there was a simple notify resource put in place that each of the execs would notify and that was documented as the resource to use to notify Service["docker-${name}"] instead of directly off the image class.

I went and looked at our manifest, and as we were hooking of a similar Exec to the above (we hadn't managed to roll forward to pick up the changes landed) that's why we weren't getting impacted by this, because we worked out that needing to let some stuff update within the Docker::Image resource was not necessarily the same as the image getting updated.

Not sure what would be the nicest way to integrate this with the existing codebase

[lukebigum]

One of our Developers has come across this exact issue. It's a long piece of string... #468, #316, #297... 😄 If I read all of that correctly, the root of the problem is Docker doesn't provide an easy way to check the SHA of an image on the registry so you can compare it to local images.

We found this: https://stackoverflow.com/a/46733555, which is not pretty, prone to break in the future, plus opens up a whole bunch of other questions (How do you get an Auth token?). It might just solve this once and for all though...

If we were to go back to an onlyif style script that can figure out what the remote SHA is without downloading it, and it works for the public Docker Registry and an internal one (eg: Pulp), would that work for people?

The work is non-trivial, but I'm willing to give it a go.

Thoughts, @electrofelix and @duritong?

[electrofelix]

@lukebigum been a while since I thought about this. I suspect it would introduce issues dealing with auth so just moving the problem elsewhere. Unless there are some ruby API bindings for docker that make it possible to hook into the standard creds and get a response without retrieving the image. I did see a few about, but not sure if they are sufficient to use here. There is definitely a python api that is supported by docker, but I'm not sure it's a good idea to introduce that as a dep.

Maybe one other way would be if a script checking for an update reset the tag back to the previous sha256 on exit as part of a trap? So the image doesn't get updated?

Other than that maybe a script to talk to the registry is one way, just expect it'll have to be able to handle auth.

Lots of ifs, buts, and maybe there. Sorry I don't see the right path so maybe your suggestion will take care of enough, and some script examples and docs for helping edge cases is the way to go

[fetzerms]

I just stumbled upon docker manifest inspect <image_name> - which seems to be "experimental", but provides what we need. Is it worth to look into adding it?

[github-actions]

This issue has been marked as stale because it has been open for a while and has had no recent activity. If this issue is still important to you please drop a comment below and we will add this to our backlog to complete. Otherwise, it will be closed in 7 days.

[ghomem]

Related to #627

[electrofelix]

Would using a custom tag to denote the last pulled image work here for all cases?

Basically retrieve the image and if tag reference is the same as the last image tagged with a last pulled tag, there is no need to notify the service to restart. Would need an additional parameter to allow control of the tag used for this. Presumably though it'll work for all docker clients instead of needing an experimental feature.