Skip to content
This repository has been archived by the owner on Nov 1, 2022. It is now read-only.

Support initContainers and {image, tag} and {image: {repository, tag}} Helm values #1258

Merged
merged 9 commits into from
Aug 2, 2018

Conversation

squaremo
Copy link
Member

@squaremo squaremo commented Jul 26, 2018

  1. Support initContainers in deployments, daemonsets etc., via
  • read and write to initContainers as well as containers in
    manifests, so we can update them;
  • report on the initContainers of Kubernetes resources
  1. Many charts use an encoding for images of
    image: image/name
    tag: tag

rather than putting the whole image ref in one field. Support that
(after kubeyaml) by interpreting the values fields, or its values,
depending on whether a "tag" entry is present.

  1. Still more charts use
    image:
      repository: org/name
      tag: v1

including those generated by helm create (and not substantially altered afterwards). Support that too.

Fixes #1226.

squaremo added 2 commits July 26, 2018 18:32
To wit:
 - update initContainers as well as containers
 - read and write {image, tag} format in FluxHelmRelease values
@squaremo squaremo requested a review from aaron7 July 26, 2018 18:49
@oliviabarrick
Copy link
Contributor

Helm charts typically use:

image:
  repository: nginx
  tag: latest

Are you intentionally using a different format than is commonly used? See #1226 for an example

@squaremo
Copy link
Member Author

Are you intentionally using a different format than is commonly used?

Argh, no, I think I just made it up in my head :-(

squaremo added 2 commits July 27, 2018 14:46
 - read and write to initContainers as well as containers in
   manifests, so we can update them;
 - report on the initContainers of Kubernetes resources
Some charts use an encoding for images of

    image: repo/name
    tag: tag

rather than putting the whole image ref in one field. Support that
(after kubeyaml) by interpreting the values fields, or its values,
depending on whether a "tag" entry is present.
@squaremo squaremo force-pushed the enhance/new-kubeyaml branch from 6c224aa to 8a00438 Compare July 27, 2018 13:48
@squaremo squaremo changed the title Support initContainers and {image, tag} Helm values Support initContainers and {image, tag} and {image: {repository, tag}} Helm values Jul 27, 2018
@squaremo
Copy link
Member Author

Hey @justinbarrick, if you get a chance, can you have a glance at this (code and or trying out)? I do not trust my head to get it right ...

@oliviabarrick
Copy link
Contributor

Your code and tests look good! You have a test that covers:

values:
  container:
    image:
      repository: blah
      tag: blah

Maybe you could cover the really basic case as well:

values:
  image:
    repository: blah
    tag: blah

And I’ll give this a try on my cluster tonight!

@oliviabarrick
Copy link
Contributor

oliviabarrick commented Jul 28, 2018

So I tried this out in my personal Kubernetes cluster, which is primarily managed using charts and FluxHelmReleases. Some notes:

  • Tags that are parsed as numbers in YAML are ignored by flux (e.g., initially this image confused it until I quoted the tag).
  • On all of my FluxHelmReleases, I'd actually only included image.tag and left off repository, since I used the default from the chart's values.yaml. Since helm-operator has you vendor the charts, typically I vendor them and modify the values.yaml to set the defaults for my cluster, then in FluxHelmReleases I only put the values that change between application instances. I'm not sure if other people do it this way and I recognize this is probably unsolvable since Flux doesn't necessarily have access to the chart repository, but figured I'd mention it.
  • A couple of third party charts do not follow any of these patterns:
  • All of my other charts (a mix of stable / incubator / random githubs / my own) appear to be compatible.
  • Upon adding the images to my FluxHelmReleases, flux upgraded the FluxHelmReleases that already had the automation annotation on them.
  • I was able to use fluxctl to automate all of the charts I've been wanting to automate.
    • However, it doesn't appear to be upgrading these images, even though it detects they are out of date (if you write the test I described above, this bug appears):
ts=2018-07-28T20:23:22.372467706Z caller=images.go:74 component=sync-loop service=media:fluxhelmrelease/jackett container=chart-image repo=linuxserver/jackett pattern=* current=linuxserver/jackett:208 info="added update to automation run" new=linuxserver/jackett:221 reason="latest 221 (2018-07-27 22:19:04.471110059 +0000 UTC) > current 208 (2018-07-07 06:59:05.611229667 +0000 UTC)"
ts=2018-07-28T20:23:22.373139255Z caller=images.go:74 component=sync-loop service=media:fluxhelmrelease/plex container=chart-image repo=plexinc/pms-docker pattern=* current=plexinc/pms-docker:1.12.3.4973-215c28d86 info="added update to automation run" new=plexinc/pms-docker:public reason="latest public (2018-07-19 18:49:15.490859368 +0000 UTC) > current 1.12.3.4973-215c28d86 (2018-04-18 20:45:18.328855429 +0000 UTC)"
ts=2018-07-28T20:23:22.37322689Z caller=images.go:74 component=sync-loop service=media:fluxhelmrelease/qbittorrent container=chart-image repo=linuxserver/qbittorrent pattern=* current=linuxserver/qbittorrent:73 info="added update to automation run" new=linuxserver/qbittorrent:87 reason="latest 87 (2018-07-27 22:32:53.488895884 +0000 UTC) > current 73 (2018-05-11 22:30:52.167200023 +0000 UTC)"
ts=2018-07-28T20:23:22.373609895Z caller=images.go:74 component=sync-loop service=media:fluxhelmrelease/radarr container=chart-image repo=linuxserver/radarr pattern=* current=linuxserver/radarr:110 info="added update to automation run" new=linuxserver/radarr:nightly reason="latest nightly (2018-07-27 22:20:13.402696345 +0000 UTC) > current 110 (2018-05-11 22:21:33.87036846 +0000 UTC)"
ts=2018-07-28T20:23:22.373682486Z caller=images.go:74 component=sync-loop service=media:fluxhelmrelease/sonarr container=chart-image repo=linuxserver/sonarr pattern=* current=linuxserver/sonarr:134 info="added update to automation run" new=linuxserver/sonarr:develop reason="latest develop (2018-07-27 22:19:19.112534675 +0000 UTC) > current 134 (2018-05-11 22:21:03.518593562 +0000 UTC)"
ts=2018-07-28T20:23:22.377940854Z caller=loop.go:102 component=sync-loop event=refreshed url=ssh://git@github.com/justinbarrick/manifests branch=master HEAD=0db4747216fd50736a5d80022641ce5e62af7311
ts=2018-07-28T20:23:22.378580004Z caller=loop.go:110 component=sync-loop jobID=2d338910-9d83-1faf-d245-da47db2c23a2 state=in-progress
ts=2018-07-28T20:23:24.272806271Z caller=warming.go:205 component=warmer fetching=justinbarrick/node-sonos-http-api-arm total=1 expired=0 missing=1
ts=2018-07-28T20:23:24.570575739Z caller=warming.go:253 component=warmer updated=justinbarrick/node-sonos-http-api-arm count=1
ts=2018-07-28T20:23:28.149576775Z caller=releaser.go:58 component=sync-loop jobID=2d338910-9d83-1faf-d245-da47db2c23a2 type=release updates=0
ts=2018-07-28T20:23:28.149670045Z caller=releaser.go:60 component=sync-loop jobID=2d338910-9d83-1faf-d245-da47db2c23a2 type=release exit="no images to update for services given"
ts=2018-07-28T20:23:28.762018678Z caller=loop.go:120 component=sync-loop jobID=2d338910-9d83-1faf-d245-da47db2c23a2 state=done success=false err="no changes made in repo"

Overall, looks great!

@oliviabarrick
Copy link
Contributor

This is a crappy patch that fixes the issue I described above: oliviabarrick@acdc513

It makes it work correctly in my cluster, but I'm not sure if that's how you want to fix it.

@oliviabarrick
Copy link
Contributor

Another case to test might be:

values:
  image:
    repository: myimage
    tag: mytag
  anothercontainer:
    image:
      repository: otherimage
      tag: othertag

@squaremo
Copy link
Member Author

This is a crappy patch that fixes the issue I described above: oliviabarrick/flux@acdc513

Ahh yeah I see the mistake I've made, thanks for puzzling that out. I wish I could think of a better way of dealing with the different types .. I'm tempted to just recursively transform the map[sinterface{}]interface{} values to map[string]interface{}. (But of course, you'd need a reference to the original value for the setter, well fiddly).

@squaremo
Copy link
Member Author

Thank you very much for checking it over @justinbarrick, that's great feedback. I'll do another round to fix up the things you pointed out.

Another case to test might be:

At present I expect one or other of the formats, whichever is seen first -- I don't think there's any reason not to accept all entries that match any format, though.

squaremo added 2 commits July 30, 2018 15:08
The output of `helm create` uses an encoding of the image to put in
the deployment template which looks like this:

    values:
      image:
        repository: weaveworks/flux
        tag: 1.4.3

This commit adds support for interpreting FluxHelmRelease resources
and manifests that use that format, as well as its derivative:

    values:
      containerA:
        image:
          repository: weaveworks/containerA
          tag: 0.1
      containerB:
        image:
          repository: weaveworks/containerB
          tag: 0.1
Support for updating FluxHelmRelease YAMLS using the {image:
{repository, tag}} format comes with kubeyaml 0.4.1.
@squaremo squaremo force-pushed the enhance/new-kubeyaml branch from 8a00438 to 77c388e Compare July 30, 2018 14:31
@squaremo
Copy link
Member Author

  • Added test for image object at the top level
values:
  image:
    repository: foo
    tag: v1

and rewrote the FluxHelmRelease values "interpretation" code to cover the missing case shown in oliviabarrick/flux@acdc513.

@oliviabarrick
Copy link
Contributor

Just tried it in my cluster and it's working well!

When intepreting the `values` part of a FluxHelmRelease, _all_
structures than can be interpreted as specifying a container
(something mentioning an image, essentially) should be returned, since
that is both more useful and more in line with expectations.

Prior to this commit, the implementation would return _either_ a
container defined at the top level (i.e., an `image` field), _or_ a
set of containers defined in other fields. But this is not adequate
even for our own flux chart.
This has the matching change to the previous commit, that is to
interpret all containers in `values` rather than a top-level `image`
field _or_ subfields.
@squaremo
Copy link
Member Author

squaremo commented Aug 1, 2018

Tags that are parsed as numbers in YAML are ignored by flux (e.g., initially this image confused it until I quoted the tag).

  • [ ] Can probably deal with this just by checking for all the possible ground types YAML can put in a field.

On all of my FluxHelmReleases, I'd actually only included image.tag and left off repository, since I used the default from the chart's values.yaml. Since helm-operator has you vendor the charts, typically I vendor them and modify the values.yaml to set the defaults for my cluster, then in FluxHelmReleases I only put the values that change between application instances. I'm not sure if other people do it this way and I recognize this is probably unsolvable since Flux doesn't necessarily have access to the chart repository, but figured I'd mention it.

Right; as it's implemented now, fluxd doesn't have access to the charts, so it won't know that the chart has a repository field that doesn't appear in the FluxHelmRelease. That's a limitation that stems from keeping the flux daemon and the helm operator separate.

You can put the repository value in the FluxHelmRelease -- a bit of a pain, and duplicates information, but a workaround.

A couple of third party charts do not follow any of these patterns:
brigade uses controller.registry and controller.name to form the image name.
stable/influxdb uses image.repo instead of image.repository.
stable/external-dns uses image.name.
All of my other charts (a mix of stable / incubator / random githubs / my own) appear to be compatible.

We could extend the interpretation of FluxHelmRelease to cover these -- I'm sure there will always be exceptions, whatever we do. I've open an issue for these varieties anyway (#1262).

Upon adding the images to my FluxHelmReleases, flux upgraded the FluxHelmReleases that already had the automation annotation on them.
I was able to use fluxctl to automate all of the charts I've been wanting to automate.
However, it doesn't appear to be upgrading these images, even though it detects they are out of date (if you write the test I described above, this bug appears):

@justinbarrick Did this problem still appear the more recent time you tried it?

EDIT: remove todo for dealing with tag values that aren't strings; see comment below.

@oliviabarrick
Copy link
Contributor

The problem I was referring to there was the one uncovered by the missing test case. So, yes, I could reproduce it but it’s resolved now 😀

@squaremo
Copy link
Member Author

squaremo commented Aug 1, 2018

Tags that are parsed as numbers in YAML are ignored by flux (e.g., initially this image confused it until I quoted the tag).

Can probably deal with this just by checking for all the possible ground types YAML can put in a field.

Arp, nope. You just have to quote values, at least those that look like floats or ints. The reason is this: YAML will parse those number-like values as ints or floats, and the actual string cannot be reliably recovered. (OK, it can if it's an int, but not if it's a float and I'd rather have no special cases).

@squaremo
Copy link
Member Author

squaremo commented Aug 1, 2018

@aaron7 With Justin's thumbs up, I reckon this is ready for code review -- sorry for the premature review request

@oliviabarrick
Copy link
Contributor

Oh good point in regards to floats! Octal and hex integers might have been an issue if you tried that too.

Copy link
Contributor

@aaron7 aaron7 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@@ -27,6 +27,11 @@ func (t PodTemplate) Containers() []resource.Container {
im, _ := image.ParseRef(c.Image)
result = append(result, resource.Container{Name: c.Name, Image: im})
}
for _, c := range t.Spec.InitContainers {
// FIXME(michael): account for possible errors here

This comment was marked as abuse.

This comment was marked as abuse.

Much better idea than naming myself.
@squaremo squaremo merged commit a61c1d5 into master Aug 2, 2018
@squaremo squaremo deleted the enhance/new-kubeyaml branch August 2, 2018 16:21
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants