Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Integration test race (or missing task reconcilier) #1500

Closed
vdemeester opened this issue Oct 30, 2019 · 3 comments
Closed

Integration test race (or missing task reconcilier) #1500

vdemeester opened this issue Oct 30, 2019 · 3 comments
Labels
area/testing Issues or PRs related to testing kind/flake Categorizes issue or PR as related to a flakey test

Comments

@vdemeester
Copy link
Member

Expected Behavior

No race.

Actual Behavior

 ---
    apiVersion: tekton.dev/v1alpha1
    kind: Task
    metadata:
      creationTimestamp: "2019-10-26T16:14:32Z"
      generation: 1
      name: status-task
      namespace: arendelle-5n7wl
      resourceVersion: "4775"
      selfLink: /apis/tekton.dev/v1alpha1/namespaces/arendelle-5n7wl/tasks/status-task
      uid: b1c92802-f80b-11e9-a134-42010a800114
    spec:
      steps:
      - args:
        - -c
        - echo hello
        command:
        - /bin/sh
        image: busybox@sha256:895ab622e92e18d6b461d671081757af7dbaa3b00e3e28e12505af7817f73649
        name: hello
        resources: {}
    
    ---
    apiVersion: tekton.dev/v1alpha1
    kind: TaskRun
    metadata:
      creationTimestamp: "2019-10-26T16:14:32Z"
      generation: 1
      name: status-taskrun
      namespace: arendelle-5n7wl
      resourceVersion: "4777"
      selfLink: /apis/tekton.dev/v1alpha1/namespaces/arendelle-5n7wl/taskruns/status-taskrun
      uid: b1cb2bff-f80b-11e9-a134-42010a800114
    spec:
      inputs: {}
      outputs: {}
      podTemplate: {}
      serviceAccountName: ""
      taskRef:
        kind: Task
        name: status-task
      timeout: 1h0m0s
    status:
      conditions:
      - lastTransitionTime: "2019-10-26T16:14:32Z"
        message: 'error when listing tasks for taskRun status-taskrun: task.tekton.dev
          "status-task" not found'
        reason: TaskRunResolutionFailed
        status: "False"
        type: Succeeded
      podName: ""
      startTime: "2019-10-26T16:14:32Z"

Steps to Reproduce the Problem

It seems to happen from time to time on the CI, on this particular test or others. We may have a race, either in the tests themselves, or we need to handle "missing" task better (aka, at least retry once if they are created at the same time and thus, the reconciler loop not seeing the task at first)

Additional Info

/area testing
@tekton-robot tekton-robot added the area/testing Issues or PRs related to testing label Oct 30, 2019
@bobcatfish
Copy link
Collaborator

image

bobcatfish added a commit to bobcatfish/plumbing that referenced this issue Oct 31, 2019
@bobcatfish
Add script for setting up GCP project permissions 🔒
1a10dbf 
In tektoncd/pipeline#1500 @vdemeester wanted
to be able to access our boskos projects to try to debug it but I had
been lazy and not given everyone access b/c there are so many of them. I
didn't want to do it (lazy) but then I realized that a script would make
it easy! So I wrote this script; it doesn't have any tests or automation
yet but eventually we could execute it as part of a Tekton pipeline and
use it to make sure permissions are always what we expect.
@bobcatfish bobcatfish mentioned this issue Oct 31, 2019
Merged
2 tasks
bobcatfish added a commit to tektoncd/community that referenced this issue Oct 31, 2019
@bobcatfish
Update boskos access
441afb8 
I had previously been lazy and not given everyone access to the boskos projects b/c it's not needed very often, but @vdemeester wanted access to help debug tektoncd/pipeline#1500 so in tektoncd/plumbing#105 I created a script to make it easy to give access to all of the 11 million boskos projects :D (and to expand the permissions if needed!)

Also added the service account permission which folks already had (to perform releases using a service account).
@bobcatfish bobcatfish mentioned this issue Oct 31, 2019
Merged
tekton-robot pushed a commit to tektoncd/plumbing that referenced this issue Nov 4, 2019
@bobcatfish @tekton-robot
Add script for setting up GCP project permissions 🔒
e3f5a4f 
In tektoncd/pipeline#1500 @vdemeester wanted
to be able to access our boskos projects to try to debug it but I had
been lazy and not given everyone access b/c there are so many of them. I
didn't want to do it (lazy) but then I realized that a script would make
it easy! So I wrote this script; it doesn't have any tests or automation
yet but eventually we could execute it as part of a Tekton pipeline and
use it to make sure permissions are always what we expect.
tekton-robot pushed a commit to tektoncd/community that referenced this issue Nov 4, 2019
@bobcatfish @tekton-robot
Update boskos access
308b4c0 
I had previously been lazy and not given everyone access to the boskos projects b/c it's not needed very often, but @vdemeester wanted access to help debug tektoncd/pipeline#1500 so in tektoncd/plumbing#105 I created a script to make it easy to give access to all of the 11 million boskos projects :D (and to expand the permissions if needed!)

Also added the service account permission which folks already had (to perform releases using a service account).
@bobcatfish
Copy link
Collaborator

Looks like this happened in #1622 as well:

      timeout: 1h0m0s
    status:
      conditions:
      - lastTransitionTime: "2019-11-26T14:42:33Z"
        message: 'error when listing tasks for taskRun status-taskrun: task.tekton.dev
          "status-task" not found'
        reason: TaskRunResolutionFailed
        status: "False"
        type: Succeeded
      podName: ""
      startTime: "2019-11-26T14:42:33Z"

@vdemeester vdemeester added the kind/flake Categorizes issue or PR as related to a flakey test label Nov 27, 2019
@bobcatfish
Copy link
Collaborator

I haven't heard anything about this so I'm going to close it and if it happens again plz reopen and link to it. Once that happens I think we should jump on it asap, I'm willing to put aside whatever other thing I'm doing and work on it!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/testing Issues or PRs related to testing kind/flake Categorizes issue or PR as related to a flakey test
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@vdemeester @bobcatfish @tekton-robot