-
Notifications
You must be signed in to change notification settings - Fork 75
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bug]: Conversion Webhook for containerservice.azure.upbound.io/v1beta1 sometimes fails #784
Comments
FWIW we saw this exact same bug with the Kubernetes provider and To me, that suggests that we are indeed hitting kubernetes/kubernetes#117356. I'd say that that if the |
Just adding here that we've seen this reappear ~3 weeks later. No obvious correlation between the errors reappearing and changes to our composition etc. |
Is there an existing issue for this?
Affected Resource(s)
kubernetesclusters.containerservice.azure.upbound.io/v1beta1
Resource MRs required to reproduce the bug
No response
Steps to Reproduce
KubernetesCluster
managed resource via a Composition Function (we're using https://github.com/crossplane-contrib/function-cue)What happened?
Occasionally the XR
Synced
status switches toFalse
temporarily due to conversion webhook error(s). This status does not propagate upwards to the claim.Relevant Error Output Snippet
And beta trace:
Crossplane Version
1.14.3
Provider Version
1.3.0
Kubernetes Version
v1.28.9
Kubernetes Distribution
AKS
Additional Info
Hi, we were previously hitting #645 and the workaround/fix was to upgrade our MR:
Since upgrading we're noticing that occasionally (and seemingly randomly) the XR
Synced
status flips toFalse
due to a conversion webhook failure. This condition only lasts for tens of seconds beforeSynced
then reverts back toTrue
. Sometimes this can take over an hour to reoccur. This status change does not propagate upwards to the claim (as shown in thecrossplane beta trace
output).I unfortunately don't have a minimal reproduction available as the environment/configuration displaying this behaviour is very complex, multiple pipeline steps etc and I haven't had much luck reproducing with an environment running in
kind
.There are no interesting logs in the Crossplane, Provider, or Function pods (even with
--debug
enabled), or the AKS control plane.My limited debugging led me to this Kubernetes bug: kubernetes/kubernetes#117356 - which seems plausible as I can see that the CRD only stores
v1beta1
:provider-upjet-azure/package/crds/containerservice.azure.upbound.io_kubernetesclusters.yaml
Line 11089 in 65f8cdc
Any ideas on where to look next?
The text was updated successfully, but these errors were encountered: