Nightly failing on TestReconcileValidDefaultWorkspace

[dibyom]

Expected Behavior

Nightly should pass

Actual Behavior

Nightly fails on the following unit test:

=== RUN   TestReconcileValidDefaultWorkspace
2020-10-23T02:12:39.153Z	ERROR	TestReconcileValidDefaultWorkspace	taskrun/controller.go:58	Failed to create taskrun metrics recorder running_taskruns_count: cannot register view "running_taskruns_count"; a different view with the same name is already registered
taskruns_pod_latency: cannot register view "taskruns_pod_latency"; a different view with the same name is already registered
github.com/tektoncd/pipeline/pkg/reconciler/taskrun.NewController.func1
	/workspace/src/github.com/tektoncd/pipeline/pkg/reconciler/taskrun/controller.go:58
github.com/tektoncd/pipeline/pkg/reconciler/taskrun.getTaskRunController
	/workspace/src/github.com/tektoncd/pipeline/pkg/reconciler/taskrun/taskrun_test.go:293
github.com/tektoncd/pipeline/pkg/reconciler/taskrun.TestReconcileValidDefaultWorkspace
	/workspace/src/github.com/tektoncd/pipeline/pkg/reconciler/taskrun/taskrun_test.go:2555
testing.tRunner
	/usr/local/go/src/testing/testing.go:1123
2020-10-23T02:12:39.156Z	INFO	TestReconcileValidDefaultWorkspace	taskrun/controller.go:93	Setting up event handlers
    taskrun_test.go:2559: Creating SA default in foo
2020-10-23T02:12:39.173Z	INFO	TestReconcileValidDefaultWorkspace.config-store	configmap/store.go:154	defaults/features/artifacts config "config-defaults" config was added or updated: &config.Defaults{DefaultTimeoutMinutes:60, DefaultServiceAccount:"default", DefaultManagedByLabelValue:"tekton-pipelines", DefaultPodTemplate:(*pod.Template)(nil), DefaultCloudEventsSink:"", DefaultTaskRunWorkspaceBinding:"emptyDir: {}"}
2020-10-23T02:12:39.174Z	INFO	TestReconcileValidDefaultWorkspace.config-store	configmap/store.go:154	defaults/features/artifacts config "feature-flags" config was added or updated: &config.FeatureFlags{DisableHomeEnvOverwrite:false, DisableWorkingDirOverwrite:false, DisableAffinityAssistant:false, RunningInEnvWithInjectedSidecars:true, RequireGitSSHSecretKnownHosts:false}
2020-10-23T02:12:39.174Z	INFO	TestReconcileValidDefaultWorkspace.config-store	configmap/store.go:154	defaults/features/artifacts config "config-artifact-bucket" config was added or updated: &config.ArtifactBucket{Location:"", ServiceAccountSecretName:"", ServiceAccountSecretKey:"", ServiceAccountFieldName:"GOOGLE_APPLICATION_CREDENTIALS"}
2020-10-23T02:12:39.176Z	INFO	TestReconcileValidDefaultWorkspace.config-store	configmap/store.go:154	defaults/features/artifacts config "config-artifact-pvc" config was added or updated: &config.ArtifactPVC{Size:"5Gi", StorageClassName:""}
{"level":"error","ts":1603419159.1718261,"logger":"fallback","caller":"taskrun/taskrun.go:313","msg":"TaskRun \"test-taskrun-default-workspace\" workspaces are invalid: declared workspace \"ws1\" is required but has not been bound","stacktrace":"github.com/tektoncd/pipeline/pkg/reconciler/taskrun.(*Reconciler).prepare\n\t/workspace/src/github.com/tektoncd/pipeline/pkg/reconciler/taskrun/taskrun.go:313\ngithub.com/tektoncd/pipeline/pkg/reconciler/taskrun.(*Reconciler).ReconcileKind\n\t/workspace/src/github.com/tektoncd/pipeline/pkg/reconciler/taskrun/taskrun.go:179\ngithub.com/tektoncd/pipeline/pkg/client/injection/reconciler/pipeline/v1beta1/taskrun.(*reconcilerImpl).Reconcile\n\t/workspace/src/github.com/tektoncd/pipeline/pkg/client/injection/reconciler/pipeline/v1beta1/taskrun/reconciler.go:240\ngithub.com/tektoncd/pipeline/pkg/reconciler/taskrun.TestReconcileValidDefaultWorkspace\n\t/workspace/src/github.com/tektoncd/pipeline/pkg/reconciler/taskrun/taskrun_test.go:2569\ntesting.tRunner\n\t/usr/local/go/src/testing/testing.go:1123"}
{"level":"error","ts":1603419159.177253,"logger":"fallback","caller":"taskrun/taskrun.go:181","msg":"TaskRun prepare error: declared workspace \"ws1\" is required but has not been bound","stacktrace":"github.com/tektoncd/pipeline/pkg/reconciler/taskrun.(*Reconciler).ReconcileKind\n\t/workspace/src/github.com/tektoncd/pipeline/pkg/reconciler/taskrun/taskrun.go:181\ngithub.com/tektoncd/pipeline/pkg/client/injection/reconciler/pipeline/v1beta1/taskrun.(*reconcilerImpl).Reconcile\n\t/workspace/src/github.com/tektoncd/pipeline/pkg/client/injection/reconciler/pipeline/v1beta1/taskrun/reconciler.go:240\ngithub.com/tektoncd/pipeline/pkg/reconciler/taskrun.TestReconcileValidDefaultWorkspace\n\t/workspace/src/github.com/tektoncd/pipeline/pkg/reconciler/taskrun/taskrun_test.go:2569\ntesting.tRunner\n\t/usr/local/go/src/testing/testing.go:1123"}
{"level":"error","ts":1603419159.205781,"logger":"fallback","caller":"taskrun/reconciler.go:294","msg":"Returned an error","targetMethod":"ReconcileKind","targetMethod":"ReconcileKind","error":"1 error occurred:\n\t* declared workspace \"ws1\" is required but has not been bound\n\n","stacktrace":"github.com/tektoncd/pipeline/pkg/client/injection/reconciler/pipeline/v1beta1/taskrun.(*reconcilerImpl).Reconcile\n\t/workspace/src/github.com/tektoncd/pipeline/pkg/client/injection/reconciler/pipeline/v1beta1/taskrun/reconciler.go:294\ngithub.com/tektoncd/pipeline/pkg/reconciler/taskrun.TestReconcileValidDefaultWorkspace\n\t/workspace/src/github.com/tektoncd/pipeline/pkg/reconciler/taskrun/taskrun_test.go:2569\ntesting.tRunner\n\t/usr/local/go/src/testing/testing.go:1123"}
    taskrun_test.go:2570: Expected no error reconciling valid TaskRun but got 1 error occurred:
        	* declared workspace "ws1" is required but has not been bound
        
    taskrun_test.go:2580: Expected TaskRun to pass Validation by using the default workspace but it did not. Final conditions were:
        v1beta1.Conditions{apis.Condition{Type:"Succeeded", Status:"False", Severity:"", LastTransitionTime:apis.VolatileTime{Inner:v1.Time{Time:time.Time{wall:0xbfdcace5ca8ff3ec, ext:23682581759, loc:(*time.Location)(0x3f92e80)}}}, Reason:"TaskRunValidationFailed", Message:"declared workspace \"ws1\" is required but has not been bound"}}
--- FAIL: TestReconcileValidDefaultWorkspace (0.06s)

Steps to Reproduce the Problem

Additional Info

https://dashboard.dogfooding.tekton.dev/#/namespaces/default/pipelineruns/pipeline-release-nightly-db2dh?pipelineTask=unit-tests&step=unit-test&view=logs

[ghost]

hmm. We can see the Default TaskRun Workspace Binding has been correctly updated (DefaultTaskRunWorkspaceBinding:"emptyDir: {}"):

2020-10-23T02:12:39.173Z	INFO	TestReconcileValidDefaultWorkspace.config-store	configmap/store.go:154	defaults/features/artifacts config "config-defaults" config was added or updated: &config.Defaults{DefaultTimeoutMinutes:60, DefaultServiceAccount:"default", DefaultManagedByLabelValue:"tekton-pipelines", DefaultPodTemplate:(*pod.Template)(nil), DefaultCloudEventsSink:"", DefaultTaskRunWorkspaceBinding:"emptyDir: {}"}

I notice the timestamp of the error (1603419159.1718261) translates to 2020-10-23T02:12:39 which is the same second that the store update messages are seen. I wonder if this was a race?

Running the same test locally it passes on master. Hmmm.