Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

puppet --noop fix has broken ensure=>latest #316

Closed
mleklund opened this issue Aug 20, 2018 · 10 comments · Fixed by #468
Closed

puppet --noop fix has broken ensure=>latest #316

mleklund opened this issue Aug 20, 2018 · 10 comments · Fixed by #468
Assignees

Comments

@mleklund
Copy link

What you expected to happen?

I expect docker::image to pull the latest image and update containers when ensure => 'latest' is set.

What happened?

Images are not being updated on linux. This looks to be introduced along with the #297 fix. The unless argument will always return 0 for latest image, so /usr/local/bin/update_docker_image.sh will never be run.

How to reproduce it?

Set ensure => latest on linux and watch nothing happen.

Anything else we need to know?

Versions:

$ puppet --version
4.10.9
$ docker version
Client:
 Version:           18.06.0-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        0ffa825
 Built:             Wed Jul 18 19:10:22 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.0-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       0ffa825
  Built:            Wed Jul 18 19:08:26 2018
  OS/Arch:          linux/amd64
  Experimental:     false
$ facter os
{
  architecture => "amd64",
  distro => {
    codename => "trusty",
    description => "Ubuntu 14.04.5 LTS",
    id => "Ubuntu",
    release => {
      full => "14.04",
      major => "14.04"
    }
  },
  family => "Debian",
  hardware => "x86_64",
  name => "Ubuntu",
  release => {
    full => "14.04",
    major => "14.04"
  },
  selinux => {
    enabled => false
  }
}
$ puppet module list
We use statically installed modules

Logs:

@mleklund
Copy link
Author

This seems to be a damned if you do, damned if you don't situation. Without running the docker pull you cannot know that the latest image has changed.

@mleklund mleklund changed the title Windows support has broken image latest puppet --noop fix has broken ensure=>latest Aug 20, 2018
@davejrt davejrt self-assigned this Aug 21, 2018
@davejrt
Copy link
Contributor

davejrt commented Aug 23, 2018

I haven't been able to replicate this issue. You can see from my logs below that a newer image with the tag latest was downloaded onto my test environment.

root@node-01:~# docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
puppet/kubetool     latest              640686af274d        18 hours ago        249MB
==> node-01: Debug: /Stage[main]/Docker/before: before to Docker::Image[puppet/kubetool]
==> node-01: Debug: /Stage[main]/Docker/before: before to Docker::Image[puppet/kubetool]
==> node-01: Debug: /Stage[main]/Docker/before: before to Docker::Image[puppet/kubetool]
==> node-01: Debug: /Stage[main]/Docker::Repos/before: before to Class[Docker::Install]
==> node-01: Debug: /Stage[main]/Docker::Repos/Apt::Source[docker]/before: before to Package[docker]
==> node-01: Debug: /Stage[main]/Apt::Update/Exec[apt_update]/before: before to Package[cgroup-lite]
==> node-01: Debug: /Stage[main]/Apt::Update/Exec[apt_update]/before: before to Package[apparmor]
==> node-01: Debug: /Stage[main]/Apt/File[sources.list]/notify: notify to Class[Apt::Update]
==> node-01: Debug: /Stage[main]/Apt/File[sources.list.d]/notify: notify to Class[Apt::Update]
==> node-01: Debug: /Stage[main]/Apt/File[preferences]/notify: notify to Class[Apt::Update]
==> node-01: Debug: /Stage[main]/Apt/File[preferences.d]/notify: notify to Class[Apt::Update]
==> node-01: Debug: /Stage[main]/Apt/File[/etc/apt/auth.conf]/notify: notify to Class[Apt::Update]
==> node-01: Debug: /Stage[main]/Docker::Install/before: before to Class[Docker::Config]
==> node-01: Debug: /Stage[main]/Docker::Config/before: before to Class[Docker::Service]
==> node-01: Debug: /Stage[main]/Docker::Service/File[/etc/systemd/system/docker.service.d/service-overrides.conf]/before: before to Service[docker]
==> node-01: Debug: /Stage[main]/Docker::Service/File[/etc/systemd/system/docker.service.d/service-overrides.conf]/notify: notify to Exec[docker-systemd-reload-before-service]
==> node-01: Debug: /Stage[main]/Docker::Service/Exec[docker-systemd-reload-before-service]/before: before to Service[docker]
==> node-01: Debug: /Stage[main]/Docker::Service/File[/etc/default/docker-storage]/notify: notify to Service[docker]
==> node-01: Debug: /Stage[main]/Docker::Service/File[/etc/default/docker]/notify: notify to Service[docker]
==> node-01: Debug: /Stage[main]/Docker::Repos/Apt::Source[docker]/Apt::Key[Add key: 9DC858229FC7DD38854AE2D88D81803C0EBFCD88 from Apt::Source docker]/before: before to Apt::Setting[list-docker]
==> node-01: Debug: /Stage[main]/Apt/Apt::Setting[conf-update-stamp]/File[/etc/apt/apt.conf.d/15update-stamp]/notify: notify to Class[Apt::Update]
==> node-01: Debug: /Stage[main]/Run_container/Docker::Image[puppet/kubetool]/Exec[echo 'Update of puppet/kubetool:latest complete']/require: require to File[/usr/local/bin/update_docker_image.sh]
==> node-01: Debug: /Stage[main]/Docker::Repos/Apt::Source[docker]/Apt::Setting[list-docker]/File[/etc/apt/sources.list.d/docker.list]/notify: notify to Class[Apt::Update]
==> node-01: Debug: /Stage[main]/Docker::Repos/Apt::Source[docker]/Apt::Key[Add key: 9DC858229FC7DD38854AE2D88D81803C0EBFCD88 from Apt::Source docker]/Apt_key[Add key: 9DC858229FC7DD38854AE2D88D81803C0EBFCD88 from Apt::Source docker]/before: before to Anchor[apt_key 9DC858229FC7DD38854AE2D88D81803C0EBFCD88 present]
==> node-01: Debug: /Stage[main]/Docker::Service/File[/etc/systemd/system/docker.service.d/service-overrides.conf]: Adding autorequire relationship with File[/etc/systemd/system/docker.service.d]
==> node-01: Debug: /Stage[main]/Docker::Repos/Apt::Source[docker]/Apt::Setting[list-docker]/File[/etc/apt/sources.list.d/docker.list]: Adding autorequire relationship with File[sources.list.d]
==> node-01: Debug: /Stage[main]/Docker::Repos/Apt::Pin[docker]/Apt::Setting[pref-docker]/File[/etc/apt/preferences.d/docker.pref]: Adding autorequire relationship with File[preferences.d]
==> node-01: Debug: Prefetching apt resources for package
==> node-01: Debug: Executing '/usr/bin/dpkg-query -W --showformat '${Status} ${Package} ${Version}\n''
==> node-01: Debug: /Stage[main]/Apt/File[/etc/apt/auth.conf]: Nothing to manage: no ensure and the resource doesn't exist
==> node-01: Debug: Prefetching apt_key resources for apt_key
==> node-01: Debug: Executing: '/usr/bin/apt-key adv --list-keys --with-colons --fingerprint --fixed-list-mode'
==> node-01: Debug: Executing: '/bin/systemctl is-active docker'
==> node-01: Debug: Executing: '/bin/systemctl is-enabled docker'
==> node-01: Debug: Exec[echo 'Update of puppet/kubetool:latest complete'](provider=posix): Executing check '/usr/local/bin/update_docker_image.sh puppet/kubetool:latest'
==> node-01: Debug: Executing: '/usr/local/bin/update_docker_image.sh puppet/kubetool:latest'
==> node-01: Debug: /Stage[main]/Run_container/Docker::Image[puppet/kubetool]/Exec[echo 'Update of puppet/kubetool:latest complete']/onlyif: latest: Pulling from puppet/kubetool
==> node-01: Debug: /Stage[main]/Run_container/Docker::Image[puppet/kubetool]/Exec[echo 'Update of puppet/kubetool:latest complete']/onlyif: ab7e51e37a18: Already exists
==> node-01: Debug: /Stage[main]/Run_container/Docker::Image[puppet/kubetool]/Exec[echo 'Update of puppet/kubetool:latest complete']/onlyif: 442b117d3b3d: Already exists
==> node-01: Debug: /Stage[main]/Run_container/Docker::Image[puppet/kubetool]/Exec[echo 'Update of puppet/kubetool:latest complete']/onlyif: 7f1f0257edc6: Already exists
==> node-01: Debug: /Stage[main]/Run_container/Docker::Image[puppet/kubetool]/Exec[echo 'Update of puppet/kubetool:latest complete']/onlyif: 65d6b2a929c4: Already exists
==> node-01: Debug: /Stage[main]/Run_container/Docker::Image[puppet/kubetool]/Exec[echo 'Update of puppet/kubetool:latest complete']/onlyif: b0c23a941824: Pulling fs layer
==> node-01: Debug: /Stage[main]/Run_container/Docker::Image[puppet/kubetool]/Exec[echo 'Update of puppet/kubetool:latest complete']/onlyif: ff2828941f57: Pulling fs layer
==> node-01: Debug: /Stage[main]/Run_container/Docker::Image[puppet/kubetool]/Exec[echo 'Update of puppet/kubetool:latest complete']/onlyif: ced286de81c2: Pulling fs layer
==> node-01: Debug: /Stage[main]/Run_container/Docker::Image[puppet/kubetool]/Exec[echo 'Update of puppet/kubetool:latest complete']/onlyif: 1a2a05a8ef62: Pulling fs layer
==> node-01: Debug: /Stage[main]/Run_container/Docker::Image[puppet/kubetool]/Exec[echo 'Update of puppet/kubetool:latest complete']/onlyif: 1a2a05a8ef62: Waiting
==> node-01: Debug: /Stage[main]/Run_container/Docker::Image[puppet/kubetool]/Exec[echo 'Update of puppet/kubetool:latest complete']/onlyif: ced286de81c2: Verifying Checksum
==> node-01: Debug: /Stage[main]/Run_container/Docker::Image[puppet/kubetool]/Exec[echo 'Update of puppet/kubetool:latest complete']/onlyif: ced286de81c2: Download complete
==> node-01: Debug: /Stage[main]/Run_container/Docker::Image[puppet/kubetool]/Exec[echo 'Update of puppet/kubetool:latest complete']/onlyif: 1a2a05a8ef62: Verifying Checksum
==> node-01: Debug: /Stage[main]/Run_container/Docker::Image[puppet/kubetool]/Exec[echo 'Update of puppet/kubetool:latest complete']/onlyif: 1a2a05a8ef62: Download complete
==> node-01: Debug: /Stage[main]/Run_container/Docker::Image[puppet/kubetool]/Exec[echo 'Update of puppet/kubetool:latest complete']/onlyif: ff2828941f57: Verifying Checksum
==> node-01: Debug: /Stage[main]/Run_container/Docker::Image[puppet/kubetool]/Exec[echo 'Update of puppet/kubetool:latest complete']/onlyif: ff2828941f57: Download complete
==> node-01: Debug: /Stage[main]/Run_container/Docker::Image[puppet/kubetool]/Exec[echo 'Update of puppet/kubetool:latest complete']/onlyif: b0c23a941824: Verifying Checksum
==> node-01: Debug: /Stage[main]/Run_container/Docker::Image[puppet/kubetool]/Exec[echo 'Update of puppet/kubetool:latest complete']/onlyif: b0c23a941824: Download complete
==> node-01: Debug: /Stage[main]/Run_container/Docker::Image[puppet/kubetool]/Exec[echo 'Update of puppet/kubetool:latest complete']/onlyif: b0c23a941824: Pull complete
==> node-01: Debug: /Stage[main]/Run_container/Docker::Image[puppet/kubetool]/Exec[echo 'Update of puppet/kubetool:latest complete']/onlyif: ff2828941f57: Pull complete
==> node-01: Debug: /Stage[main]/Run_container/Docker::Image[puppet/kubetool]/Exec[echo 'Update of puppet/kubetool:latest complete']/onlyif: ced286de81c2: Pull complete
==> node-01: Debug: /Stage[main]/Run_container/Docker::Image[puppet/kubetool]/Exec[echo 'Update of puppet/kubetool:latest complete']/onlyif: 1a2a05a8ef62: Pull complete
==> node-01: Debug: /Stage[main]/Run_container/Docker::Image[puppet/kubetool]/Exec[echo 'Update of puppet/kubetool:latest complete']/onlyif: Digest: sha256:e87151f10477f4eaf5927b1e42c4dab0cb2091c97451c240289ce4124eb064bb
==> node-01: Debug: /Stage[main]/Run_container/Docker::Image[puppet/kubetool]/Exec[echo 'Update of puppet/kubetool:latest complete']/onlyif: Status: Downloaded newer image for puppet/kubetool:latest
==> node-01: Debug: /Stage[main]/Run_container/Docker::Image[puppet/kubetool]/Exec[echo 'Update of puppet/kubetool:latest complete']/onlyif: puppet/kubetool:latest updated. Changed from sha256:640686af274d2e77b6c23ddb5a0b1fba2667242daad8e310e55af295de70cf46 to sha256:d803d1a398b1ad8783adaf3565419a98c00911cf78ac1a9e823db9094bffb323.
==> node-01: Debug: Exec[echo 'Update of puppet/kubetool:latest complete'](provider=posix): Executing 'echo 'Update of puppet/kubetool:latest complete''
==> node-01: Debug: Executing: 'echo 'Update of puppet/kubetool:latest complete''
==> node-01: Notice: /Stage[main]/Run_container/Docker::Image[puppet/kubetool]/Exec[echo 'Update of puppet/kubetool:latest complete']/returns: Update of puppet/kubetool:latest complete
==> node-01: Notice: /Stage[main]/Run_container/Docker::Image[puppet/kubetool]/Exec[echo 'Update of puppet/kubetool:latest complete']/returns: executed successfully
==> node-01: Debug: /Stage[main]/Run_container/Docker::Image[puppet/kubetool]/Exec[echo 'Update of puppet/kubetool:latest complete']: The container Docker::Image[puppet/kubetool] will propagate my refresh event
==> node-01: Debug: Docker::Image[puppet/kubetool]: The container Class[Run_container] will propagate my refresh event
==> node-01: Debug: Class[Run_container]: The container Stage[main] will propagate my refresh event
==> node-01: Debug: Finishing transaction 36490880
==> node-01: Debug: Storing state
==> node-01: Debug: Stored state in 0.01 seconds
==> node-01: Notice: Applied catalog in 24.51 seconds
==> node-01: Debug: Applying settings catalog for sections reporting, metrics
==> node-01: Debug: Finishing transaction 29507000
==> node-01: Debug: Received report to process from node-01.gateway
==> node-01: Debug: Evicting cache entry for environment 'production'
==> node-01: Debug: Deleted text domain :production: false
==> node-01: Debug: Caching environment 'production' (ttl = 0 sec)
==> node-01: Debug: Processing report from node-01.gateway with processor Puppet::Reports::Store

root@node-01:~# docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
puppet/kubetool     latest              d803d1a398b1        26 minutes ago      249MB
puppet/kubetool     <none>              640686af274d        19 hours ago        249MB
root@node-01:~#

@mleklund
Copy link
Author

Are you running with the patch from #297, i.e. master? I do not see your code running the unless.

Aug 20 11:28:24 host puppet-agent[119833]: (Exec[/usr/local/bin/update_docker_image.sh docker.domain.net/ops/varnish-proxy:latest](provider=posix)) Executing check 'docker images -q docker.domain.net/ops/varnish-proxy:latest | grep .'
Aug 20 11:28:24 host puppet-agent[119833]: Executing: 'docker images -q docker.domain.net/ops/varnish-proxy:latest | grep .'
Aug 20 11:28:24 host puppet-agent[119833]: (/Stage[main]/Profile::Varnish::Proxy/Docker::Image[docker.domain.net/ops/varnish-proxy]/Exec[/usr/local/bin/update_docker_image.sh docker.domain.net/ops/varnish-proxy:latest]/unless) 836453e62df4

I do not see how the unless check docker images -q docker.domain.net/ops/varnish-proxy:latest | grep . can ever return anything other then 0 if with the grep . on there. Further, I am not sure you can come up with a test to satisfy the unless clause without actually running the docker pull to check your registry, there is no way for docker image ls to test if there is a new image.

I realize I am running a non-release version, but am doing that because of the upstart bug from #304.

@davejrt
Copy link
Contributor

davejrt commented Aug 24, 2018

@mleklund you're correct in that I was running the code from the forge, and not what was in master and I can replicate your issue.

I am not aware of a test that is going to be simple, without having to introduce more params and logic and using things like curl to get information out of the registry. I'll open this up for discussion internally as well but it's possible we may have to consider a breaking change in the next release as using the latest tag is generally considered a poor practice, no matter how practical it may be in dev scenarios.

We will however be happy to review a PR that can cleanly address the issue.

@davejrt
Copy link
Contributor

davejrt commented Nov 27, 2018

Closing as we have acknowledged this as a breaking change. We are happy to review PR's that attempt to address the issue without adding too much complexity to the use of the module

@electrofelix
Copy link
Contributor

@davejrt Is there a reason that the noop metaparameter cannot be used in this case?

@davejrt
Copy link
Contributor

davejrt commented Mar 13, 2019

noop can be used with the module now, without any execs being runs. Previously if you ran with noop would still check the registry for later images and pull them if they were available...which as was agreed in the previous issue, not a noop operation.

@electrofelix
Copy link
Contributor

@davejrt I meant if the metaparameter noop => false was set on the old exec resource, it would skipped during a noop run while marking the resource as having been refreshed and logging that any subscribed resources would have received a refresh event, while still resulting in the resource being executed as expected for a real run.

See https://puppet.com/docs/puppet/6.0/metaparameter.html#noop

What I was wondering was there a particular reason that use of this metaparameter on the exec resource definition instead of removing the behaviour wasn't a valid option for this use case?

@davejrt
Copy link
Contributor

davejrt commented Mar 13, 2019

I'm not sure if this was tested and didn't work, but we will be open to reviewing any PR's that address the issue.

@electrofelix
Copy link
Contributor

@davejrt good to know, I'll have a look at putting something together if someone else doesn't get to it first, just wanted to check there wasn't an obvious reason it wasn't used.

electrofelix added a commit to electrofelix/puppetlabs-docker that referenced this issue Apr 3, 2019
Restore the code to update images tagged with 'latest' on each run,
moving this to be guarded by a refreshonly meta-parameter that is
triggered by a notify resource that is disabled for 'noop'.

Because the original code to update images was run via 'onlyif' to
indicate if the image was in sync by performing a pull and testing the
new digest against the old, this meant that simply applying 'noop' to
this exec resource would not work. 'noop' does not affect running the
components of a resource to validate if it is in-sync, which is the
purpose of the 'onlyif' & 'unless' settings for an exec.

To support 'noop' mode, previously the code to update the image on each
run was dropped to prevent it accidentally refreshing the local image
and then skipping an image restart because there was no update event on
a subsequent real run.

To resolve, it is necessary to have a prior resource that is used as a
proxy, and only when it notifies the image update resource, will the
'onlyif' action be tested to determine if in sync. This prior resource
can be disabled from a noop run, and subsequently although puppet
records that it would send a refresh event in 'noop' mode, none are
actually sent during a 'noop' run thus preventing the image from being
updated by a side effect of checking if the notified exec was in-sync.

Fixes puppetlabs#316
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants