Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

The recording webhook's resource updating is racy #744

Closed
jhrozek opened this issue Dec 2, 2021 · 6 comments · Fixed by #1112
Closed

The recording webhook's resource updating is racy #744

jhrozek opened this issue Dec 2, 2021 · 6 comments · Fixed by #1112
Labels
kind/bug Categorizes issue or PR as related to a bug. lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale.

Comments

@jhrozek
Copy link
Contributor

jhrozek commented Dec 2, 2021

What happened:

I deleted pods in a deployment to trigger recording a policy and saw:

Error from server: admission webhook "recording.spo.io" denied the request: failed to update resource profilerecording: Operation cannot be fulfilled on profilerecordings.security-profiles-operator.x-k8s.io "nginx-selinux-policy": the object has been modified; please apply your changes to the latest version and try again
Error from server: admission webhook "recording.spo.io" denied the request: update resource on removing pod: failed to update resource profilerecording status: Operation cannot be fulfilled on profilerecordings.security-profiles-operator.x-k8s.io "nginx-selinux-policy": the object has been modified; please apply your changes to the latest version and try again
Error from server: admission webhook "recording.spo.io" denied the request: update resource on removing pod: failed to update resource profilerecording status: Operation cannot be fulfilled on profilerecordings.security-profiles-operator.x-k8s.io "nginx-selinux-policy": the object has been modified; please apply your changes to the latest version and try again

What you expected to happen:

No errors

How to reproduce it (as minimally and precisely as possible):

  • create a profilerecording
  • create a largish deployment (I managed to reproduce with about ~50 pods) matching the profilerecording
  • kubectl delete the pods in the deployment

Anything else we need to know?:

N/A

Environment:

  • Cloud provider or hardware configuration: AWS
  • OS (e.g: cat /etc/os-release): RHCOS 4.9
  • Kernel (e.g. uname -a): N/A
  • Others: N/A
@jhrozek jhrozek added the kind/bug Categorizes issue or PR as related to a bug. label Dec 2, 2021
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle stale
  • Mark this issue or PR as rotten with /lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 2, 2022
@jhrozek
Copy link
Contributor Author

jhrozek commented Mar 2, 2022

/remove-lifecycle stale

@k8s-ci-robot k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 2, 2022
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle stale
  • Mark this issue or PR as rotten with /lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label May 31, 2022
@jhrozek
Copy link
Contributor Author

jhrozek commented Jun 1, 2022

/remove-lifecycle stale

@k8s-ci-robot k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jun 1, 2022
jhrozek added a commit to jhrozek/security-profiles-operator that referenced this issue Aug 23, 2022
@jhrozek
webhooks: Make setting finalizers and status more resilient with retries
9eac582 
Especially with multiple replicas, we've seen the webhooks error out due
to a conflict. This is problematic because the webhooks have a hard-fail
policy.

Let's retry multiple times instead.

Fixes: kubernetes-sigs#744
@jhrozek jhrozek mentioned this issue Aug 23, 2022
Merged
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle stale
  • Mark this issue or PR as rotten with /lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Aug 30, 2022
@jhrozek
Copy link
Contributor Author

jhrozek commented Aug 30, 2022

/remove-lifecycle stale
this has a PR on review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug. lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

recording webhook: Remove webhook state jhrozek/security-profiles-operator
3 participants
@jhrozek @k8s-ci-robot @k8s-triage-robot