Immutable field detection

[gdziwoki]

Before CEL transition rules was a thing, CRDs would use custom admission webhooks for CRD validation.
I have a Kustomization that manages a TargetGroupBinding
Kustomization has force: true but when I change an immutable field in the TargetGroupBinding I get an error on my Kustomization syncs:

TargetGroupBinding/kube-system/haproxy-ingress-tgb dry-run failed, reason: TargetGroupBinding update may not change these fields: spec.targetGroupARN, error: admission webhook "vtargetgroupbinding.elbv2.k8s.aws" denied the request: TargetGroupBinding update may not change these fields: spec.targetGroupARN

Is there any other way to force kustomize-controller to recreate those resources?

My env:
Kubernetes EKS 1.24
kustomize-controller v0.31.0

[stefanprodan]

We detect immutable errors using Kubernetes errors package:

https://github.com/fluxcd/pkg/blob/3a4ad50c09b68efe348532efe393743e82df518c/ssa/utils.go#L287-L295

func IsImmutableError(err error) bool {
	// Detect immutability like kubectl does
	// https://github.com/kubernetes/kubectl/blob/8165f83007/pkg/cmd/apply/patcher.go#L201
	if errors.IsConflict(err) || errors.IsInvalid(err) {
		return true
	}
	return false
}

The webhook response doesn't match IsInvalid, so there is nothing we can do about it in Flux.

Can you confirm that kubectl apply --server-side --force fails the same?

If kubectl fails with --force, please open an issue in Kubernetes for them to make the CEL validation error conformant to the Kubernetes conventions. Or maybe they can expose a new errors helper to detect CEL immutable errors.

[gdziwoki]

It fails with the following error:

$ kubectl apply --server-side --force-conflicts -f tgb.yaml
Error from server (TargetGroupBinding update may not change these fields: spec.targetGroupARN): admission webhook "vtargetgroupbinding.elbv2.k8s.aws" denied the request: TargetGroupBinding update may not change these fields: spec.targetGroupARN

[stefanprodan]

@gdziwoki going to close this issue, we've added immutable error detection to Flux when using CEL or any custom webhook that errors out with field is immutable. This doesn't fix the current issue but you can switch to using CEL or modify the TargetGroupBinding webhook to return:

spec.targetGroupARN is immutable