PipelineRun fails too eagerly on missing resources

[mattmoor]

Expected Behavior

PipelineRun only fails when referenced resources do not exist.

Actual Behavior

PipelineRun fails when referenced resources do not YET exist in it's informer cache at the time of reconciliation.

You can see the bug here as the lister is passed to fetch the resource, and on error it manifests as a permanent failure of the pipeline resource:

pipeline/pkg/reconciler/pipelinerun/pipelinerun.go

Lines 334 to 341 in 1710b68

	providedResources, err := resources.GetResourcesFromBindings(pr, c.resourceLister.PipelineResources(pr.Namespace).Get)
	if err != nil {
	// This Run has failed, so we need to mark it as failed and stop reconciling it
	pr.Status.MarkFailed(ReasonCouldntGetResource,
	"PipelineRun %s/%s can't be Run; it tries to bind Resources that don't exist: %s",
	pipelineMeta.Namespace, pr.Name, err)
	return controller.NewPermanentError(err)
	}

Steps to Reproduce the Problem

This is an intermittent flake in the e2e testing, which manifests with a message like:

PipelineRun arendelle-8mqgj/git-check-pipeline-run can''t be Run; it tries to bind Resources that don''t exist: error following resource reference for git-repo: pipelineresource.tekton.dev "git-source-resource" not found

Additional Info

This is roughly at HEAD (I observed this downstream, but the issue is clearly upstream)

[mattmoor]

As @jlpettersson pointed out in slack, they had to put the PipelineRun at the bottom here for the test to pass:
https://github.com/tektoncd/pipeline/blob/master/examples/v1beta1/pipelineruns/pipelinerun-using-different-subpaths-of-workspace.yaml

IMO a good signal that this is solid would be that this passes with the PipelineRun first. It's a decent simulation of informer lag, and aims for a better experience for users that might not want to deal with orchestrating multiple applies.

[mattmoor]

Slack thread: https://tektoncd.slack.com/archives/CLCCEBUMU/p1602598837131500

[GregDritschler]

This is very closely related to #2740. The pipelinerun and taskrun reconcilers use the client to get the Pipeline or Task respectively to avoid lister cache issues. There was a lot of good discussion in that issue. I'm not sure why it was closed.

[mattmoor]

There is a lot of nuance, and honestly there are facets of the Tekton resource model that make this challenging (as I highlighted in that thread).

The conflict here is between:

1. fast-feedback when a user fat-fingers a bad reference
2. working in the presence of slow informer syncing

Using the client could solve both, but then puts things into conflict with scaling the system.

I think an appropriate solution here would be to give the informer caches a grace period before declaring missing references fatal. This could key off of metadata.creationTimestamp.

[mattmoor]

I have a change, which will hopefully build in a grace period for this here: #3385

Let's see how the e2e tests respond to this 🤞

[LiYa-Xu]

Hi all ,I am not sure I met a same issue. The e2e test case fails on case TestTaskRunPipelineRunCancel, I got message like

CONT  TestTaskRunPipelineRunCancel/retries=1
  build_logs.go:34: Could not get logs for pod cancel-me-task-nzln8-pod-ddrn7: container "step-unnamed-0" in pod "cancel-me-task-nzln8-pod-ddrn7" is waiting to start: PodInitializing
    build_logs.go:36: build logs 

Then I checked the events from this e2e test namespces , seems the pod is still under pending status:

arendelle-hpcnl    19m   Normal    Created         pod/cancel-me-task-mwvc9-pod-qn26j                     Created container place-scripts
arendelle-hpcnl    19m   Normal    Started         pod/cancel-me-task-mwvc9-pod-qn26j                     Started container place-scripts
arendelle-hpcnl    19m   Normal    Started         taskrun/cancel-me-task-mwvc9                           
arendelle-hpcnl    19m   Normal    Pending         taskrun/cancel-me-task-mwvc9                           Pending
arendelle-hpcnl    19m   Normal    Pending         taskrun/cancel-me-task-mwvc9                           pod status "Initialized":"False"; message: "containers with incomplete status: [place-scripts place-tools]"
arendelle-hpcnl    19m   Warning   Failed          taskrun/cancel-me-task-mwvc9                           TaskRun "cancel-me-task-mwvc9" was cancelled
arendelle-hpcnl    19m   Normal    Started         pipelinerun/cancel-me                                  
arendelle-hpcnl    19m   Normal    Running         pipelinerun/cancel-me                                  Tasks Completed: 0 (Failed: 0, Cancelled 0), Incomplete: 1, Skipped: 0
arendelle-hpcnl    19m   Warning   Failed          pipelinerun/cancel-me                                  PipelineRun "cancel-me" was cancelled

Can you help me?