Server-Side Apply support

[skolyszewski]

Hi!

We just hit a nasty issue related to client-side apply and using last-applied-configuration annotation. If spec in some manifest is too long, the request is rejected, because annotations have size limit, and l-a-c is basically the spec. Server-side apply deals with that, because there's no longer a need to use l-a-c, as far as I understand it.
From what I see and understand, this provider implements client-side apply approach only. Do you have any thoughts or plans wrt. supporting server-side apply? We may be able to contribute, but since it's a big feature (both options would have to co-exist, so that's probably a lot of new code or branches in CRUD), we figured to ask that question first.

Best,
Grzegorz

[pst]

I'm definitely interested in server-side applies. I have not looked into it yet though. Supporting both side by side, with server side default and client side fallback is one option. Depending on how well server side applies work and how far back in K8s versions they are supported, another option could be to make a future version of the provider server side apply only. And users that only have client side support on their K8s clusters have to stay with the old provider version. First step would probably need to be, to get a picture of what it would take to implement server side applies.

[skolyszewski]

We've done more research and it turns out that it's beyond difficult to use server-side apply when trying to apply multiple manifests in one run, as supported by design in this provider. We've tried feeding manifests rendered by kustomization_build into Kubernetes provider's kubernetes_manifest which uses server-side apply and it doesn't work, exactly because of above.
There's also a problem of two sides fighting for control over the state (tf state and kube control plane) which is non-trivial.

Instead, we've decided to "work-around" the issue and shorten the manifest of the faulty resource and it works without issues.
For the context: the faulty resource was Prometheus CRD from kube-prometheus stack, version 0.10. We've just created a patch that has description fields emptied, which makes it half as long and possible to be applied using client-side apply.

[pst]

The provider does not send multiple resources to the K8s API at the same time. The data source returns a map(string). The kustomization_resource has for_each selecting one of the map entries by id. Terraform then calls the provider once per instance of the kustomization_resource. Any methods inside the provider only handle a single K8s resource in JSON format at a time.

Using for_each there should be no reason why the JSON manifests returned by the data source couldn't be fed into kubernetes_manifest wrapped in jsondecode.

[skolyszewski]

The provider does not send multiple resources to the K8s API at the same time.

Of course. What I said was that the provider applies multiple manifests in a single terraform run, because this is what kustomize is for, on a high level (unless you kustomize a single manifest).

The reason for failures is that during planning phase, if CRDs don't exist and you're trying to apply CRs - it fails. You must split CRDs and CRs into multiple terraform runs, which is technically possbile thanks to priority calculation in the provider, but definitely far from being a good solution.

Here's a ref: hashicorp/terraform-provider-kubernetes#1380

[pst]

Yes, so providers have no control over the graph. But the solution I have for the kustomization provider is, that the data sources provide id_prio and by looping over the ids according to their prio and having a depends_on that makes prio 1 depend on prio 0 and prio 2 on prio 1 you can apply the CRD and CO reliably in one go. Since the prio is coming out of the data source and the depends_on is built-in you can do the same using kubernetes_manifest instead of kustomization_resource.

https://registry.terraform.io/providers/kbst/kustomization/latest/docs/resources/resource#explicit-depends_on-example