Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix duplicate TaskRuns when Pipeline/PipelineRun labels are changed #3558

Merged
merged 1 commit into from
Dec 1, 2020

Conversation

GregDritschler
Copy link
Contributor

@GregDritschler GregDritschler commented Nov 23, 2020

Fixes #3126

The PipelineRun reconciler may be handed a PipelineRun that does not have a previous status update containing the names of TaskRuns it created. The PipelineRun reconciler attempts to deal with this by looking at the TaskRun lister cache for TaskRuns associated with the PipelineRun that are missing from the status and adding them. It uses the PipelineRun's labels to find them. If the user changes the PipelineRun's labels (or the Pipeline's labels which are propagated to the PipelineRun) while the pipeline is running, it may cause the code to not find existing TaskRuns and as a result it may duplicate them.

Changes

The reconciler is changed to use only the tekton.dev/pipelineRun label when listing TaskRuns associated with the PipelineRun.

/kind bug

Submitter Checklist

These are the criteria that every PR should meet, please check them off as you
review them:

  • Includes tests (if functionality changed/added)
  • Includes docs (if user facing)
  • Commit messages follow commit message best practices
  • Release notes block has been filled in or deleted (only if no user facing changes)

See the contribution guide for more details.

Double check this list of stuff that's easy to miss:

Reviewer Notes

If API changes are included, additive changes must be approved by at least two OWNERS and backwards incompatible changes must be approved by more than 50% of the OWNERS, and they must first be added in a backwards compatible way.

Release Notes

Fix an issue where the PipelineRun controller could create duplicate TaskRuns if a Pipeline's or PipelineRun's labels are changed while the PipelineRun is running

@tekton-robot tekton-robot added release-note-none Denotes a PR that doesnt merit a release note. kind/bug Categorizes issue or PR as related to a bug. labels Nov 23, 2020
@tekton-robot tekton-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Nov 23, 2020
@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/reconciler/pipelinerun/pipelinerun.go 84.4% 84.6% 0.2

@vdemeester
Copy link
Member

@GregDritschler I think we need a release-note entry though 👼

@tekton-robot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vdemeester

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@tekton-robot tekton-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-none Denotes a PR that doesnt merit a release note. labels Nov 24, 2020
Copy link
Member

@afrittoli afrittoli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this, nice catch and fix!
/lgtm

t.Fatalf("unexpected error when updating status: %v", err)
}

// The label update triggers another reconcile. Depending on timing, the PipelineRun passed to the reconcile may or may not
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice trick :)

@vdemeester
Copy link
Member

/hold

@tekton-robot tekton-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 24, 2020
…re changed

The PipelineRun reconciler may be handed a PipelineRun that does not have a previous status update containing the names of TaskRuns it created.
The PipelineRun reconciler attempts to deal with this by looking at the TaskRun lister cache for TaskRuns associated with the PipelineRun
that are missing from the status and adding them.  It uses the PipelineRun's labels to find them.  If the user changes the PipelineRun's labels
(or the Pipeline's labels which are propagated to the PipelineRun) while the pipeline is running, it may cause the code to not find existing
TaskRuns and as a result it may duplicate them.
@tekton-robot tekton-robot removed the lgtm Indicates that a PR is ready to be merged. label Nov 24, 2020
@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/reconciler/pipelinerun/pipelinerun.go 84.4% 84.6% 0.2

@ghost
Copy link

ghost commented Nov 30, 2020

/lgtm

Thanks for digging into this @GregDritschler !

@tekton-robot tekton-robot assigned ghost Nov 30, 2020
@tekton-robot tekton-robot added the lgtm Indicates that a PR is ready to be merged. label Nov 30, 2020
@vdemeester
Copy link
Member

/hold cancel

@tekton-robot tekton-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 1, 2020
@tekton-robot tekton-robot merged commit 73d30ed into tektoncd:master Dec 1, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. kind/bug Categorizes issue or PR as related to a bug. lgtm Indicates that a PR is ready to be merged. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Executing a Pipeline with a single Task results in multiple TaskRuns
5 participants