Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix for ResourceQuotaConflictError #5252

Merged
merged 1 commit into from
Aug 31, 2022
Merged

Fix for ResourceQuotaConflictError #5252

merged 1 commit into from
Aug 31, 2022

Conversation

yachna
Copy link
Contributor

@yachna yachna commented Aug 2, 2022 •
edited by dibyom
Loading

Changes

Fix for the bug ResourceQuotaConflictError kubernetes/kubernetes#67761

From @SaschaSchwarze0

We face the problem that when you create a resource (for example a Pod) that is constrained by a resource quota, then the creation can fail because of conflicts while updating the resource quota status (that's the long-standing Kubernetes issue that @yachna mentioned, kubernetes/kubernetes#67761).

Submitter Checklist

As the author of this PR, please check off the items in this checklist:

  • Has Docs included if any changes are user facing
  • Has Tests included if any functionality added or changed
  • Follows the commit message standard
  • Meets the Tekton contributor standards (including
    functionality, content, code)
  • Has a kind label. You can add one by adding a comment on this PR that contains /kind <type>. Valid types are bug, cleanup, design, documentation, feature, flake, misc, question, tep
  • Release notes block below has been updated with any user facing changes (API changes, bug fixes, changes requiring upgrade notices or deprecation warnings)
  • Release notes contains the string "action required" if the change requires additional action from users switching to the new release

Release Notes

Tekton will retry the creation of the Pod if it fails due to a conflict and results in ResourceQuotaConflictError while updating a ResourceQuota.

@tekton-robot tekton-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Aug 2, 2022
@linux-foundation-easycla
Copy link

linux-foundation-easycla bot commented Aug 2, 2022 •
edited
Loading

CLA Signed

The committers listed above are authorized under a signed CLA.

  • ✅ login: yachna / name: Yachna (f37c173)

@tekton-robot tekton-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Aug 2, 2022
@tekton-robot
Copy link
Collaborator

Hi @yachna. Thanks for your PR.

I'm waiting for a tektoncd member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@SaschaSchwarze0
Copy link
Contributor

/kind bug

@tekton-robot tekton-robot added the kind/bug Categorizes issue or PR as related to a bug. label Aug 2, 2022
@yachna yachna marked this pull request as ready for review August 3, 2022 04:54
@tekton-robot tekton-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 3, 2022
@dibyom
Copy link
Member

dibyom commented Aug 3, 2022

Shouldn't this already already be handled in

pipeline/pkg/reconciler/taskrun/taskrun.go

Line 577 in 94055d9

case isExceededResourceQuotaError(err):
?

Also see #734

@SaschaSchwarze0
Copy link
Contributor

Shouldn't this already already be handled in

pipeline/pkg/reconciler/taskrun/taskrun.go

Line 577 in 94055d9

case isExceededResourceQuotaError(err):

?

Also see #734

Hi @dibyom, we are not facing an exceeded resource quota. We face the problem that when you create a resource (for example a Pod) that is constrained by a resource quota, then the creation can fail because of conflicts while updating the resource quota status (that's the long-standing Kubernetes issue that @yachna mentioned, kubernetes/kubernetes#67761).

But maybe that function is where we need to move the retry logic @yachna, with maybe a requeue after one second.

@dibyom
Copy link
Member

dibyom commented Aug 4, 2022

@SaschaSchwarze0 thanks for the explanation! and yeah, instead of implementing custom retry logic, I think we should catch the error and put it back in the work queue to retry.

@dibyom
Copy link
Member

dibyom commented Aug 4, 2022

/ok-to-test

@tekton-robot tekton-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Aug 4, 2022
@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/reconciler/taskrun/taskrun.go 80.6% 78.9% -1.7

@tekton-robot tekton-robot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Aug 9, 2022
@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/reconciler/taskrun/taskrun.go 80.6% 80.9% 0.3

@yachna yachna marked this pull request as draft August 9, 2022 07:01
@tekton-robot tekton-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Aug 9, 2022
@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/reconciler/taskrun/taskrun.go 80.6% 81.4% 0.8

SaschaSchwarze0
SaschaSchwarze0 suggested changes Aug 9, 2022
View reviewed changes
pkg/reconciler/taskrun/taskrun.go Outdated Show resolved Hide resolved
@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/reconciler/taskrun/taskrun.go 80.6% 81.1% 0.5

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/reconciler/taskrun/taskrun.go 80.6% 81.1% 0.5

@linux-foundation-easycla
Copy link

linux-foundation-easycla bot commented Aug 10, 2022 •
edited
Loading

CLA Signed

The committers listed above are authorized under a signed CLA.

  • ✅ login: yachna / name: Yachna (3bb201f)

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/reconciler/taskrun/taskrun.go 80.3% 80.8% 0.5

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/reconciler/taskrun/taskrun.go 80.3% 80.8% 0.5

@tekton-robot tekton-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Aug 11, 2022
@SaschaSchwarze0
Copy link
Contributor

@dibyom we are ready here, code has been moved :-)

@dibyom
Copy link
Member

dibyom commented Aug 11, 2022

Thanks @yachna and @SaschaSchwarze0 The code looks fine to me. One request - could we update the commit message with a bit more description around what the issue was that this commit solves (like the description here so that it meets our commit guidelines: https://github.com/tektoncd/community/blob/main/standards.md#commits

@tekton-robot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vdemeester

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@tekton-robot tekton-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 18, 2022
@yachna
Copy link
Contributor Author

yachna commented Aug 22, 2022

Thanks @yachna and @SaschaSchwarze0 The code looks fine to me. One request - could we update the commit message with a bit more description around what the issue was that this commit solves (like the description here so that it meets our commit guidelines: https://github.com/tektoncd/community/blob/main/standards.md#commits

Hi @dibyom

Thanks for the suggestion. I have updated the commit message accordingly.

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/reconciler/taskrun/taskrun.go 80.4% 80.9% 0.5

@yachna
Fix for the bug ResourceQuotaConflictError
eef94f6 
This commit contains fix for the bug ResourceQuotaConflictError More details about the issue are here kubernetes/kubernetes#67761
@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/reconciler/taskrun/taskrun.go 80.4% 80.9% 0.5

@dibyom
Copy link
Member

dibyom commented Aug 30, 2022

/lgtm

@tekton-robot tekton-robot added the lgtm Indicates that a PR is ready to be merged. label Aug 30, 2022
@SaschaSchwarze0
Copy link
Contributor

/test pull-tekton-pipeline-integration-tests
/test pull-tekton-pipeline-alpha-integration-tests

@abayer
Copy link
Contributor

abayer commented Aug 31, 2022

/retest

@tekton-robot tekton-robot merged commit 05f28f2 into tektoncd:main Aug 31, 2022
@yachna yachna deleted the patch-2 branch September 5, 2022 02:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@SaschaSchwarze0 SaschaSchwarze0 SaschaSchwarze0 requested changes

@vdemeester vdemeester vdemeester approved these changes

@abayer abayer Awaiting requested review from abayer

@dibyom dibyom Awaiting requested review from dibyom

@lbernick lbernick Awaiting requested review from lbernick

@dlorenc dlorenc Awaiting requested review from dlorenc

@imjasonh imjasonh Awaiting requested review from imjasonh

Assignees

@dibyom dibyom

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. kind/bug Categorizes issue or PR as related to a bug. lgtm Indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@yachna @tekton-robot @SaschaSchwarze0 @dibyom @abayer @vdemeester