Ability to apply annotations for entire environment.

[captncraig]

I have a few potential use cases where it would be helpful to add an annotation to an entire environment, perhaps for information or controlling some other part of our process. Primarily I'd like to be able to add fluxcd.io/ignore: true to every resource in an environment in an easy way. Also possibly an ability to add something like the prune labels, but for our own auditing with cd, rather than for native tanka pruning.

My desire would be to add annotations to the environment spec:

{
  "apiVersion": "tanka.dev/v1alpha1",
  "kind": "Environment",
  "metadata": {
    "name": "blahblahblah",
    "labels": {
      "team": "whatever@example.com"
    }
  },
  "spec": {
    "apiServer": "https://127.0.0.1",
    "injectLabels": true,
    "namespace": "cert-manager",
    "annotations": {
      "fuxcd.io/ignore": "true",
      "tanka.dev/environment": "blahblahblah",
    }
  }
}

This would be pretty trivial to implement alongside the existing injectLabels logic. I will have a pr for that shortly.

A few considerations to debate:

• We could do labels in the same way. Is there desire or use case to do that now?
• But it may be confusing having too many similar looking concepts in the spec. Metadata labels, injectLabels field, annotations and/or labels in the spec.

[issue-label-bot]

Issue-Label Bot is automatically applying the label kind/feature to this issue, with a confidence of 0.95. Please mark this comment with 👍 or 👎 to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[captncraig]

As discussed today in tanka call, this is not necessarily our favorite, since it does more and more "magic" outside of what the jsonnet declares.

Let's consider some alternatives:

1. Native jsonnet. Make a function that can be invoked from main.jsonnet that can traverse your other manifests and inject annotations. Or maybe patch the kubernetes library with something magical.
2. Post-processing. Do it in export step or in some other process on the exported manifests from tk.
3. A command line flag to tk apply that injects annotations one time only.

The native jsonnet way is interesting since it doesn't require tanka changes. But I fear it will be hard to implement with objects at different nesting levels potentially. And it could be a little tricky for environment authors to integrate if they aren't sure what they are doing. With the spec.json, it is pretty easy to make changes, even in an emergency.

I started playing with option 2 a bit, but realized one workflow I want to support is disabling flux and making a change by setting the flag and running tk apply. Requiring the cd flow to override the cd flow is a little counterproductive.

Number 3 is intriguing, but I don't see the point when I think about it. Tanka still has to implement the logic, and the user doesn't get the advantage of being able to save the state in git. It feels like punishing the user for no great reason.

In short, I can see how this adds complexity in a way we might not necessarily love, but it is a tiny amount of code that works. And all the alternatives I've considered so far have pain in them.

[chancez]

What about supporting a more general hooks-like solution that allows patching the resources after they're rendered, where the input is a list containing all the resources post-transformation (no hidden fields, just JSON), and you transform it using jsonnet.

You could write a jsonnet function to add annotations to everything, and make it a re-usable import. This would also let someone write other transformations they want to apply to everything. I feel like the environment file spec.injectLabels field could even be re-implemented with this approach.

This would also be similar to the kustomize transformers concept.

[Duologic]

Native jsonnet. Make a function that can be invoked from main.jsonnet that can traverse your other manifests and inject annotations. Or maybe patch the kubernetes library with something magical.

Something like this?
https://github.com/grafana/jsonnet-libs/blob/2bead07b1497283ce4e3b3fd7f2c0b141a973a13/helm-util/helm.libsonnet#L43-L62

[captncraig]

Nice @Duologic. What would be the best way to integrate that into main.jsonnet for all objects? If easy enough that could work.

[captncraig]

I am also intrigued by a "hooks" system. It might have philosophical conflicts with how we feel jsonnet should work, but it would be very powerful to have project-level external transforms that get run on everything. So I could write a piece of code that looks at the environment metadata labels and applies various transforms or checks dynamically. That is probably another discussion though.

[Duologic]

Nice @Duologic. What would be the best way to integrate that into main.jsonnet for all objects? If easy enough that could work.

Good question, you would need to wrap this around 'everything' to make it work, but maybe someone can find a more clever way to do this.

[sh0rez]

Implemented in 073e616