Ability to include other type of source in GitRepository

[davinkevin]

Hello Flux Team 👋,

We are using currently the include attribute in GitRepository to bypass some limitations (especially remove base with git) and for path reconciliation, but the current API allows us to use only GitRepository into another GitRepository, like this:

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: GitRepository
metadata:
  name: foo-app-source
  namespace: flux-system
spec:
  interval: 99m
  include:
    - repository:
        name: fo-experiment-base
      fromPath: foor/.base
      toPath: apps/cluster/foo/.base
  ref:
    branch: 'main'
  url: ssh://git@github.com/foo/bar.git
  secretRef:
    name: flux-system
---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: GitRepository
metadata:
  name: foo-experiment-base
  namespace: flux-system
spec:
  interval: 99m
  ref:
    tag: 'foo-1.8'
  url: ssh://git@github.com/another/repo-with-base-only.git
  secretRef:
    name: another-secret

And, we would like to be able to use something else than repository in include, for example:

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: GitRepository
metadata:
  name: foo-app-source
  namespace: flux-system
spec:
  interval: 99m
  include:
    - source:
        apiVersion: source.toolkit.fluxcd.io/v1beta2
        kind: Bucket
        name: foo-s3-base
      fromPath: foor/.base
      toPath: apps/cluster/foo/.base
  ref:
    branch: 'main'
  url: ssh://git@github.com/foo/bar.git
  secretRef:
    name: flux-system

Or even, in (near) future, being able to do the same with OCI registry like this:

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: GitRepository
metadata:
  name: foo-app-source
  namespace: flux-system
spec:
  interval: 99m
  include:
    - source:
        apiVersion: source.toolkit.fluxcd.io/v1beta2
        kind: OCIRepository
        name: foo-oci-base
      fromPath: foor/.base
      toPath: apps/cluster/foo/.base
  ref:
    branch: 'main'
  url: ssh://git@github.com/foo/bar.git
  secretRef:
    name: flux-system

If you need more feedback or use case we have from that, feel free to ask, I'll be happy to share in more details our structure.

PS: Flux is a super product, thank you for it!

[stefanprodan]

I think that with the OCI features that are coming next in Flux, there is no more need for include, as you can piece together the desired configuration from various sources in a CI job and publish the final manifests to a container registry with flux push artifact.

[davinkevin]

Hello @stefanprodan and thank you for you answer.

I've done a lot of investigation about kustomizer and the OCI feature currently (I've read the RFC) in progress in Flux and to be honest, I don't see it as a solution for it at the moment. I detail that:

As a manifest editor, I would like to publish a set of YAML, the base (Kustomization & Components) alongside my containers images. But, I clearly don't know how people will use this base, what components they will enable, and what are the configuration. So a base without configuration is useless 🤷‍♂️.

As a consumer, I would like to keep simplicity of the current workflow (Kustomize+GitRepository)… and I think, from my understanding it will be more complicated with OCI features because:

• I can't store secrets in OCI entry, for obvious security reasons
• ☝️ because of previous point, I will have to disable disableNameSuffixHash because kustomize won't be able to determine the name of the secret beforehand
• ☝️ will trigger the same thing for ConfigMap
• Won't be able to use immutable configmap anymore (for audit purpose)
• most important of all, I'll have to publish the whole set of manifests in advance for a given environment, so setup a complete CI pipeline and everything related for every env… even ephemeral ones (because I don't have access to the OCI registry as publisher)
• A configuration change will require a CI build which is way slower than flux + webhook (magnitude x100)

As an operator of a potential OCI registry, we don't want to have 1 entry in this registry for each combination of base times configuration possible. In helm, we only manage product version and it's already a lot 😅.

And finally, as an administrator, I use flux to provide to many teams ability to install and configure any system on demand. They just need to provide some configuration in a kustomization.yaml/values.yaml to configure the installation. With the current OCIRepository solution, this won't be possible to use a it as a base without doing the publication process before…

I follow this subject for a long time now (since the first release of kustomizer I've tried day 1) and I've discussed a lot with kustomize users since (I've done a lot of conference about it where I recommend flux BTW) and they massively use kustomize with remote base/components, usually with git::. base is published by one team and consumed by another one… without extra build steps in between. The current solution for me feels like if we had to do helm template … > foo.yaml && git add . && git push to support helmRelease in flux, we would loose the simplicity… and I feel the same with current OCI solution as a Kustomize user.

Sorry for this really too long answer, I really wanted to share my concerns about the OCI feature, from an end-user perspective… who really like Flux! If you want, I'm available for a quick call to discuss and share feedbacks on this subject.

And thank you again for the great work made in flux 🤩 !

[stefanprodan]

I can't store secrets in OCI entry, for obvious security reasons

Yes you can, OCI is no different from Git, so use SOPS.

Won't be able to use immutable configmap

Yes you can, like before "OCI is no different from Git".

They just need to provide some configuration in a kustomization.yaml/values.yaml to configure the installation. With the current OCIRepository solution, this won't be possible to use a it as a base without doing the publication process before…

For a HelmChart you provide the config with HelmRelease.spec.values/valuesFrom, for OCIRepository you'll do the same but with Kustomization.spec.patches and/or Kustomization.spec.postBuild.

[davinkevin]

Thank you @stefanprodan for your answer.

You're right, secrets/configMap/Immutability can be done but is not compatible with system where you store base manifests and configuration in two different places. It's really a common usage, especially when you have multiple envs.

For a HelmChart you provide the config with HelmRelease.spec.values/valuesFrom, for OCIRepository you'll do the same but with Kustomization.spec.patches and/or Kustomization.spec.postBuild.

Sorry, but inlined Kustomization.spec.patches & Kustomization.spec.postBuild is not the same as an original kustomization.yaml.

Here, you have a (really standard) kustomization.yaml we use:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ../.base
  - monitoring
  - ingress

components:
  - liveness-probe-modifications
  - node-scheduling
  - ../.base/components/application-features/foo-bar
  - ../.base/components/ha
  - ../.base/components/hpa

images:
  - name: my/app
    newName: gcr.io/davinkevin/foo/bar

configMapGenerator:
  - behavior: merge
    literals:
      - a=one
      - b=two
      - c=three
    name: application

The associated structure in my IDE to make this clearer:

FYI, monitoring, ingress are kustomize.config.k8s.io/v1beta1/Kustomization too importing some components from the base too with specific replacements.

And this would be for me the equivalent of values / valuesFrom in HelmRelease. And again, from my personal/professional experience, it's a common way to publish / install / configure an application with Kustomize.

[stefanprodan]

Thanks @davinkevin for the detailed explanations.

Would a new API type better solve this issue? something like:

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: Bundle
metadata:
  name: foo-app-source
  namespace: flux-system
spec:
  include:
    - source:
        apiVersion: source.toolkit.fluxcd.io/v1beta2
        kind: OCIRepository
        name: foo-base
      fromPath: deploy
      toPath: apps/cluster/foo/base
    - source:
        apiVersion: source.toolkit.fluxcd.io/v1beta2
        kind: GitRepository
        name: foo-config
      fromPath: overlays/production
      toPath: apps/cluster/foo/production

[davinkevin]

If I understand it correctly, a bundle is a set of different source mixed together and it can be referenced as a source from a kustomization?

It looks good to me. We then remove the hierarchy between base and overlays in my previous example 👍. If I'm not wrong, in our case it can reduce drastically the number of flux object, because we have currently for each deployment:

• 2 flux.GitRepository (one base, one overlay)
• 1 flux.Kustomization

With this, we would be able to have:

• 1 flux.GitRepository shared accros all deployments (our overlays are in a single repository)
• 1 OCIRepository for each deployment (some attached to latest, some to a tag).

So, if I'm doing the math right, it could be way better for us 😇.

Out of curiosity, do you have an idea about the reconciliation strategy attached to a bundle? And, is a bundle meant to receive notification(s), or just included objects?

On the paper, like this, it looks really good to me 😉, thank you for sharing this! If you want to discuss about our use case and our installation, feel free to ping me 😉. In the meantime, I've described it in a blogpost recently.

[adawalli]

@stefanprodan - your proposed API change is exactly what we need as well. In our deployments, we wish to deploy a base set of resources to our teams, but allow them to provide kustomize patches (e.g., add in their own ARNs, annotations, etc) - I think your proposal above would provide exactly the fix we need

[mlschindler]

+1 to @stefanprodan's proposed solution. Provides exactly the solution we need to provide base layers to our teams and they are free to add overlays on top without concerning themselves with extra sources.

[souleb]

For OCIRepository, could we solve this by building a multi-layered OCI artifact?

We would retrieve base and components in CI, add our overlays, and finally build/push to our registry. Flux would then make sure to provide an artifact that contains all the layers merged.

[mlschindler]

Keeping this on everyone's radar. Any traction on creating a new Bundle API?

[skurtzemann]

@stefanprodan have you some news about the new Bundle API you proposed?

[ghost]

@stefanprodan +1 on this proposed API, it sounds really great. I was reading the gitrepo include source code trying to figure out what happens if the toPath already exists in the base source (a merge or complete replacement), and came across this. This would be really great! I'm trying to come up with a way to version my kustomization bases such that when I make changes to them I don't roll them out to all my clusters all at once. this would be similar to Terraform's ability to choose module versions from gitrepo references etc.

[ghost]

Cross-namespace references allowed pls 🙏 🙏 🙏

[hiddeco]

Cross-namespace references allowed pls

We have at present no plans to further support cross-namespace references anywhere in the Flux APIs. See also fluxcd/flux2#2092 (comment)

[ghost]

Cross-namespace references allowed pls

We have at present no plans to further support cross-namespace references anywhere in the Flux APIs. See also fluxcd/flux2#2092 (comment)

@hiddeco Skimming through the ReferenceGrant proposal mentioned in that comment, it sounds like something equivalent, is that right?

[mlschindler]

Keeping this on everyone's radar. Any traction on creating a new Bundle API?

Yearly bump for updates on Bundle API 😄

[maximveksler]

This would be really useful for us. Our use case is reusing 1 bucket for multiple flux environments that need to be isolated from the other.

So we have apps directory that should be shared, and clusters/<clusterName> path uniquely available to each flux env using AWS ACCESS_KEY.

Since the IAM policy locks access tighthly, and Bucket CRD does not support multiple paths in prefix a work around could have been 2 Bucket configruations. Pointing at the same bucket with one with prefix apps and the other with prefix clusters/<clusterName> than merged using the mentioned spec.