ApisixTls status not updated #1221

papdaniel · 2022-08-04T19:00:40Z

Issue description

I use cert manager to generate certificates and create the Certificate and ApisixTls resources at the same time. Cert-manager needs some time to request the certificate, so when apisix-ingress-controller tries to find the cert secret it doesn't exist yet. The problem is apisix-ingress-controller won't retry to find the secret, so we are stucked. I know it is documented here that the secret should exist when we create the ApisixTls resource, but with a retry mechanism it would be much more relieable.

Edit: as I see, there is a retry rule in apisix-ingress-controller, but in this case it doesn't seem to retry. Maybe because of the ResourceSyncAborted status reason?

Environment

your apisix-ingress-controller version (output of apisix-ingress-controller version --long):

Version: 1.4.1
Git SHA: no-git-module
Go Version: go1.16.15
Building OS/Arch: linux/amd64
Running OS/Arch: linux/amd64

your Kubernetes cluster version (output of kubectl version):

Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.1", GitCommit:"3ddd0f45aa91e2f30c70734b175631bec5b5825a", GitTreeState:"clean", BuildDate:"2022-05-24T12:26:19Z", GoVersion:"go1.18.2", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"22+", GitVersion:"v1.22.11-eks-18ef993", GitCommit:"b9628d6d3867ffd84c704af0befd31c7451cdc37", GitTreeState:"clean", BuildDate:"2022-07-06T18:06:23Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

Minimal test code / Steps to reproduce

Install cert-manager and apisix-ingress-controller, and
create the following:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: test
  namespace: default
spec:
  dnsNames:
  - '*.test.example.com'
  issuerRef:
    group: cert-manager.io
    kind: ClusterIssuer
    name: letsencrypt-prod
  secretName: test-wildcard
---
apiVersion: apisix.apache.org/v2beta3
kind: ApisixTls
metadata:
  name: test
  namespace: default
spec:
  hosts:
  - '*.test.example.com'
  secret:
    name: test-wildcard
    namespace: default

Actual result

status:
  conditions:
  - lastTransitionTime: "2022-08-04T13:23:20Z"
    message: secret "test-wildcard" not found
    observedGeneration: 1
    reason: ResourceSyncAborted
    status: "False"
    type: ResourcesAvailable

Error log

Expected result

No response

The text was updated successfully, but these errors were encountered:

papdaniel · 2022-08-05T07:14:04Z

Ehh, my bad, it works as expected, the status tricked me, it is not updated.

tao12345666333 · 2022-08-05T14:30:54Z

Ehh, my bad, it works as expected, the status tricked me, it is not updated.

Do you mean the status of the ApisixTls resource is not updated?
@papdaniel

papdaniel · 2022-08-06T02:22:15Z

Yes @tao12345666333

tao12345666333 · 2022-08-06T05:58:31Z

We expected that all status should be updated correctly, so I reopened this issue

tao12345666333 · 2022-08-06T06:00:42Z

Will reproduce when I'm free, if this is confirmed we can fix it

Can you provide a minimal complete reproduction steps if you are interested?

papdaniel · 2022-08-08T12:30:31Z

@tao12345666333 sure, I will on the weekend if that's fine, don't have time right now

papdaniel · 2022-08-14T12:35:22Z

Here is how to reproduce:

create namespace and the ApisixTls:

kubectl create namespace apisix-tls-test

cat <<EOF | kubectl apply -f -
apiVersion: apisix.apache.org/v2beta3
kind: ApisixTls
metadata:
  name: test-tls
  namespace: apisix-tls-test
spec:
  hosts:
  - "*.example.com"
  secret:
    name: test-tls
    namespace: apisix-tls-test
EOF

check the ApisixTls status:

kubectl -n apisix-tls-test get apisixtls test-tls -o yaml

...
status:
  conditions:
  - lastTransitionTime: "2022-08-14T19:01:48Z"
    message: secret "test-tls" not found
    observedGeneration: 1
    reason: ResourceSyncAborted
    status: "False"
    type: ResourcesAvailable
...

ingress controller logs (secret not found):

kubectl -n ingress-apisix logs <apisix-ingress-controller-pod>
...
022-08-14T19:02:46+08:00       warn    ingress/apisix_tls.go:178       sync ApisixTls failed, will retry       {"object": {"Type":1,"Object":"apisix-tls-test/test-tls","Tombstone":null}, "error": "secret \"test-tls\" not found"}
2022-08-14T19:02:46+08:00       error   ingress/apisix_tls.go:118       failed to translate ApisixTls   {"error": "secret \"test-tls\" not found", "ApisixTls": {"metadata":{"name":"test-tls","namespace":"apisix-tls-test","uid":"44895291-29d5-4454-a011-079a64140420","resourceVersion":"49712","generation":1,"creationTimestamp":"2022-08-14T11:02:46Z","annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apisix.apache.org/v2beta3\",\"kind\":\"ApisixTls\",\"metadata\":{\"annotations\":{},\"name\":\"test-tls\",\"namespace\":\"apisix-tls-test\"},\"spec\":{\"hosts\":[\"*.example.com\"],\"secret\":{\"name\":\"test-tls\",\"namespace\":\"apisix-tls-test\"}}}\n"},"managedFields":[{"manager":"apisix-ingress-controller","operation":"Update","apiVersion":"apisix.apache.org/v2beta3","time":"2022-08-14T11:02:46Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{".":{},"f:conditions":{}}},"subresource":"status"},{"manager":"kubectl-client-side-apply","operation":"Update","apiVersion":"apisix.apache.org/v2beta3","time":"2022-08-14T11:02:46Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}}},"f:spec":{".":{},"f:hosts":{},"f:secret":{".":{},"f:name":{},"f:namespace":{}}}}}]},"spec":{"hosts":["*.example.com"],"secret":{"name":"test-tls","namespace":"apisix-tls-test"}},"status":{"conditions":[{"type":"ResourcesAvailable","status":"False","observedGeneration":1,"lastTransitionTime":"2022-08-14T11:02:46Z","reason":"ResourceSyncAborted","message":"secret \"test-tls\" not found"}]}}}
...

wait a few seconds, then create the tls secret

cat <<EOF | kubectl apply -f -
kind: Secret
apiVersion: v1
metadata:
  name: test-tls
  namespace: apisix-tls-test
data:
  tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURmVENDQW1XZ0F3SUJBZ0lVQlpyZ0M2ZFFTWmNWYzNhWlJlU2R6Ulg4aktJd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0ZqRVVNQklHQTFVRUF3d0xaWGhoYlhCc1pTNWpiMjB3SGhjTk1qSXdPREE0TVRJMU56QXpXaGNOTkRreApNakkwTVRJMU56QXpXakFTTVJBd0RnWURWUVFEREFkMFpYTjBMV051TUlJQklqQU5CZ2txaGtpRzl3MEJBUUVGCkFBT0NBUThBTUlJQkNnS0NBUUVBeWtuU2pJMjRkSHg5L1hxRGwwcHQ2emVydUh0N29FQXhwOHp2bW85TUp1aFQKSTQxL2NNZENCVi93TFFPQnk5TzRTMVhBcU9tUEpPUDNPQ1JsbWZPdndyYXhHY0V0NWlYNG4vT2dmRzhuMVJQLwo3emlwZG5DZWpzMVQzNFh0TFZnL3RzNWM4MDBJU0NtRGlLaUtLcUl6SktaUDVQRVB2b2tXNlV2TTVaNVNCbTJKClBiU2psL0I2MUtBVW5UaEVzSjUvc2tqckgyOVB0N1JONFVBTDNlM1RmL1YxZW9ESmZJOWo0QlRjSGdTQUNvL1QKOXBWNmo3cWs4aWdqSEJpWWtFaW1RSi9vKzRXSFhTb2IwYzdIN3lRbVBXLzJ0UFNvd2twMnlTUGhZQWRla0NBMQo1TDVHTjl6ZUl4TFJpZDVXKzd0WldkYzliU3RIWGFoQkY5OUVHTFJ4UndJREFRQUJvNEhHTUlIRE1GRUdBMVVkCkl3UktNRWlBRktnMnk0ZGlzR1BUaVhJY2lwTjUyMjNYa04wUW9ScWtHREFXTVJRd0VnWURWUVFEREF0bGVHRnQKY0d4bExtTnZiWUlVT1FMZlZqa094elQ2KzZDZEZlTGFnNHd1ZGVvd0NRWURWUjBUQkFJd0FEQUxCZ05WSFE4RQpCQU1DQkRBd0hRWURWUjBsQkJZd0ZBWUlLd1lCQlFVSEF3RUdDQ3NHQVFVRkJ3TUNNQmdHQTFVZEVRUVJNQStDCkRTb3VaWGhoYlhCc1pTNWpiMjB3SFFZRFZSME9CQllFRkZuS2hmQkhoT1JwYVB0eVF5NVBBMExmMDBtOE1BMEcKQ1NxR1NJYjNEUUVCQ3dVQUE0SUJBUUFBQ2NSK0J2S1FPYmw2YUhwZnpObE1GajVXTmhucWE1N3NVelpYREhERQpzVEpJMFlDa3hVZHBEdXNCS3RlSno1dWgwTmlXZkI5c2RSa3F1c2V6b3dvSkU0UnZnMUNqOWRrWlQ4TzZPN2VPCjljczFkM0hydEdoODMvemk5ajM3WFNldEIzQXNkUHA1ZnhjcXQzbDVLVGE4enZIUHF5ZWhxS3JHamlodkFRUGkKeUZzMGsra0R1TXIvSk96QmY2b09xS2dGdy9LVmZrTFdQSlRmSWlvNjYzcUp4dDQzTmlsbnpQNGFpY2dXQm93RgpHcXJndDdOcHhBRWkvMlEwYkF5b0JFdURuZngwamxSMytKaWtuYTNxUENMQlAzNEFqZ0ZJZnFUOGNjWjIwQm5ECk9aMitDM2xra1JhZ3luMlRtUU5RTzBXbzh5YVJLRUY1T0g5Wk5mZThIbDZICi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
  tls.key: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUV2UUlCQURBTkJna3Foa2lHOXcwQkFRRUZBQVNDQktjd2dnU2pBZ0VBQW9JQkFRREtTZEtNamJoMGZIMzkKZW9PWFNtM3JONnU0ZTN1Z1FER256TythajB3bTZGTWpqWDl3eDBJRlgvQXRBNEhMMDdoTFZjQ282WThrNC9jNApKR1daODYvQ3RyRVp3UzNtSmZpZjg2QjhieWZWRS8vdk9LbDJjSjZPelZQZmhlMHRXRCsyemx6elRRaElLWU9JCnFJb3Fvak1rcGsvazhRKytpUmJwUzh6bG5sSUdiWWs5dEtPWDhIclVvQlNkT0VTd25uK3lTT3NmYjArM3RFM2gKUUF2ZDdkTi85WFY2Z01sOGoyUGdGTndlQklBS2o5UDJsWHFQdXFUeUtDTWNHSmlRU0taQW4rajdoWWRkS2h2Ugp6c2Z2SkNZOWIvYTA5S2pDU25iSkkrRmdCMTZRSURYa3ZrWTMzTjRqRXRHSjNsYjd1MWxaMXoxdEswZGRxRUVYCjMwUVl0SEZIQWdNQkFBRUNnZ0VBQklld1JMRDlkN0J5T2c3S3JQcTNqOEdiK3Mwc0VNYzRnVDJKbDNiQXJ0SmYKRk1qbDhIL3RhWFlqVmVGQVlpMTFwdE1VdktMSTNyNzlPQ1U0R0RQUlQ0c2d3NmJ3R0hZZWwrb3dUeFFEQXdkWApRcGxrRlJNd2Rab2NROVdNSFpoUkE3QnNIVEZ6aUp0dU94NDVBUWo5bFNoRHF1d242T1QvUGNYcUdmVUY4bUVUCkRsRWtRUXowSi9BNmhqTURDR244U1Z1V29lallOdWJHRnV6OGdQUEdhUFowcjltbXlrRHBpQkMrbTluZkpjQnUKM1REWDY5Rmc1dzJSSWs4STQ4VTdVWW5yZjQ3aDRVbDA2NFU3a0VtekNYcERwdDJvQk1pMzFqWDMvajJuL2tkUApGMjNEdEY0RHgvT1FackFBMzVtaTRkZ2x5WCtQWHZTZ2hIa2dHaUF6cVFLQmdRRFZybGlONFFVbFJmTTlCU0F6Cmw3YWEySnpTNXYyTldVNVpXWlVhd1NIRFNlUERDL21LZnpLVzBOYWNvQmFkWnJ2VkJibTMrTUszbjBvVlRDaFkKUVN3c1ByZmFjV3VIRk1tejZZWFFjT0djWWVOeW8yelBRbU9LcnB0WC9Vck5GeGozRUV6QTcrelBPOUxRQXZFego1NFJWYkFRQUJoMUU5ZmFISHpORjljbjByUUtCZ1FEeVdkM29sZ2p6amlZbjgwMGJ1a0JEWUxRVEFHN2tRd1BDCjd4TFQzN21zL001S1I2d1dEVUE3VzlXWFlqcjAxbGJidm5Ha0RNdHNLM0poRlZMQ3dMN1VHWUU4dWVKZmdyNHgKOVFtcFMwN05XWXVqdC81MGdEa3VsNjRUdndIZ25JWFJTK0ZSRHE1K0I5OWtlaWkyZktwQjUrRlFUWWI4Y3VCawpNR2V5L0lRSVF3S0JnRlNSRUlqdUlrd3pGK1h4Ym1JY3krOFNoVlpPaHRCM0pMUGw2VVhTWTV0WFZlV3VpbzhPCmNpWVA5cSs4QjljNWZtTDdSaG9TUUhYTmU5TmI5YWVmKzJ4elVlNEk3MFpocXNZaVNJeEs4RzhrcXBDSEx4d0QKcVVPTWdQeGN5VXJMdGx5cFBjY3A3UzNDL2xiR28xRW9jazRnVnFEdnlvdXRVR2ovcHhvc0FVWnRBb0dBSHZ3awpaZ2RBcjdyOTM5KzRmUklNQTFGSzFKOGFqbTZPL1k1SmJ2Tk1ydlJIZjdHNEs3Mlg5NzdmQ1JMVEROaVdsRk85Ci9jWncvNnh5WEQwNHJwT3NFandNalBJTElrRDZCak1aQUI4VHZTZDB0Z0xoTW8zdVFhbTg1T05uM3dyRUJTVloKY2w2WktwM2hEdHliZGZGc1p5NjVYT1g4bXl3b3lrWnlDNjFEU0RjQ2dZRUFnK1hkWVdYTDQwNzY5MVlKSEJDTQpKcG5BZjM2V2dOUXNLVGxwbkR0MFNGNk9LRms3VjZPRXdXdFgxWm1VUmY4a1B2YWxUTFh0OEZYcVAyRk90TkRYClUzTlNNdEhxS3lWeHZwTm5YdUFLSlkrNnNtckZhdXZTTlVHWEVoUElUdi9rcXZOdGRmMXhFaml6cTJ4NGpoaDAKZkZUUEppWThoWjlORVdTWkVoaHB0RWs9Ci0tLS0tRU5EIFBSSVZBVEUgS0VZLS0tLS0K
type: kubernetes.io/tls
EOF

ingress controller logs (only healthcheck, no error):

kubectl -n ingress-apisix logs  <apisix-ingress-controller-pod>
...
[GIN] 2022/08/14 - 19:03:11 | 200 |      42.061µs |     10.0.43.201 | GET      "/healthz"
[GIN] 2022/08/14 - 19:03:11 | 200 |       36.23µs |     10.0.43.201 | GET      "/healthz"
[GIN] 2022/08/14 - 19:03:21 | 200 |      36.421µs |     10.0.43.201 | GET      "/healthz"
[GIN] 2022/08/14 - 19:03:21 | 200 |      22.711µs |     10.0.43.201 | GET      "/healthz"
...

apisix logs (ssl resource created successfully in apisix):

kubectl -n ingress-apisix logs <apisix-pod>
...
10.0.41.91 - - [14/Aug/2022:11:03:51 +0000] apisix-admin.ingress-apisix.svc.cluster.local:9180 "PUT /apisix/admin/ssl/21210e5d HTTP/1.1" 201 3878 0.049 "-" "Go-http-client/1.1" - - - "http://apisix-admin.ingress-apisix.svc.cluster.local:9180"
...

Check the ApisixTls status again, not updated:

kubectl -n apisix-tls-test get apisixtls test-tls -o yaml

...
status:
  conditions:
  - lastTransitionTime: "2022-08-14T19:01:48Z"
    message: secret "test-tls" not found
    observedGeneration: 1
    reason: ResourceSyncAborted
    status: "False"
    type: ResourcesAvailable
...

tao12345666333 · 2022-08-14T20:38:02Z

thanks!

tao12345666333 · 2022-08-25T04:15:26Z

Would you like to try to fix this problem?

There is a recordStatus function in the code to record the change of the status

papdaniel · 2022-08-28T10:49:59Z

Yes, sure,I'll try.

papdaniel · 2022-08-28T13:53:24Z

Just cloned and tried out on master, now it's working:

I think it got solved by this commit a few weeks ago: d32c728 on these and these lines.

tao12345666333 · 2022-08-29T02:29:58Z

Thanks for your confirmation, then I think this issue can be closed, right?

tao12345666333 · 2022-08-29T02:32:54Z

Also, I thought maybe we should create a patch for 1.5.0 to use .
Because we haven't released the official version of v1.5.0, the current release is the rc1 version

You can try v1.5.0 branch https://github.com/apache/apisix-ingress-controller/tree/v1.5.0
Thanks!

papdaniel · 2022-09-04T13:49:04Z

I won't really have time for this, sorry.

tao12345666333 · 2022-09-05T08:24:01Z

Don't worry, thanks for your contribution

papdaniel closed this as completed Aug 5, 2022

tao12345666333 changed the title ~~bug: ApisixTls: secret not found, should be retried~~ ApisixTls status not updated Aug 6, 2022

tao12345666333 reopened this Aug 6, 2022

tao12345666333 added the checking label Aug 10, 2022

tao12345666333 self-assigned this Aug 10, 2022

tao12345666333 added bug Something isn't working triage/accepted Indicates an issue or PR is ready to be actively worked on. labels Aug 25, 2022

tao12345666333 added this to Apache APISIX Ingress controller Aug 25, 2022

tao12345666333 moved this to Todo in Apache APISIX Ingress controller Aug 25, 2022

tao12345666333 added this to the v1.6.0 milestone Aug 25, 2022

AlinsRan mentioned this issue Sep 15, 2022

fix: crd resource status is not updated #1335

Merged

8 tasks

tao12345666333 linked a pull request Sep 16, 2022 that will close this issue

fix: crd resource status is not updated #1335

Merged

8 tasks

tao12345666333 closed this as completed in #1335 Sep 16, 2022

Repository owner moved this from Todo to Done in Apache APISIX Ingress controller Sep 16, 2022

AlinsRan mentioned this issue Feb 2, 2023

bug: Informer recursively trigger update event #682

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ApisixTls status not updated #1221

ApisixTls status not updated #1221

papdaniel commented Aug 4, 2022 •

edited

Loading

papdaniel commented Aug 5, 2022

tao12345666333 commented Aug 5, 2022

papdaniel commented Aug 6, 2022

tao12345666333 commented Aug 6, 2022

tao12345666333 commented Aug 6, 2022

papdaniel commented Aug 8, 2022

papdaniel commented Aug 14, 2022

tao12345666333 commented Aug 14, 2022

tao12345666333 commented Aug 25, 2022

papdaniel commented Aug 28, 2022

papdaniel commented Aug 28, 2022

tao12345666333 commented Aug 29, 2022

tao12345666333 commented Aug 29, 2022

papdaniel commented Sep 4, 2022

tao12345666333 commented Sep 5, 2022

ApisixTls status not updated #1221

ApisixTls status not updated #1221

Comments

papdaniel commented Aug 4, 2022 • edited Loading

Issue description

Environment

Minimal test code / Steps to reproduce

Actual result

Error log

Expected result

papdaniel commented Aug 5, 2022

tao12345666333 commented Aug 5, 2022

papdaniel commented Aug 6, 2022

tao12345666333 commented Aug 6, 2022

tao12345666333 commented Aug 6, 2022

papdaniel commented Aug 8, 2022

papdaniel commented Aug 14, 2022

tao12345666333 commented Aug 14, 2022

tao12345666333 commented Aug 25, 2022

papdaniel commented Aug 28, 2022

papdaniel commented Aug 28, 2022

tao12345666333 commented Aug 29, 2022

tao12345666333 commented Aug 29, 2022

papdaniel commented Sep 4, 2022

tao12345666333 commented Sep 5, 2022

papdaniel commented Aug 4, 2022 •

edited

Loading