Tekton metrics are reporting wrong values

[RafaeLeal]

Expected Behavior

Tekton metrics coherent with data gather with kubectl

Actual Behavior

I have two examples, for the tekton_pipelinerun_count metric, it's currently reporting 22 failed and 58 success, but the cluster only have 20 PipelineRuns

$ kubectl get pipelinerun --all-namespaces -o json | jq '.items | length'
20

The second example, tekton_pipelinerun_duration_seconds_sum metric seems to increase indefinitely. I expected to raise until the pipelinerun is completed, than stabilize in the same value as the duration showed with tkn pr describe <pipelinerun_name>

I also noticed that

1. It doesn’t raise at the same rate.
2. Always raises a multiple of the actual duration of the pipeline.

I suspect that every time the controller reconciles it’s adding the duration of the pipeline.

Steps to Reproduce the Problem

1. Enable tekton metrics
2. Make Prometheus scrape tekton
3. Run some pipelineruns

Additional Info

• Kubernetes version:

  Output of kubectl version:

  $ kubectl version
  Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-26T06:17:09Z", GoVersion:"go1.14", Compiler:"gc",Platform:"darwin/amd64"}
  Server Version: version.Info{Major:"1", Minor:"15+", GitVersion:"v1.15.11-eks-af3caf", GitCommit:"af3caf6136cd355f467083651cc1010a499f59b1", GitTreeState:"clean", BuildDate:"2020-03-27T21:51:36Z", GoVersion:"go1.12.17", Compiler:"gc", Platform:"linux/amd64"}
  

• Tekton Pipeline version: v0.13.2

  Output of tkn version

  $ tkn version
  Client version: 0.8.0
  Pipeline version: unknown
  

  Output of kubectl get pods -n tekton-pipelines -l app=tekton-pipelines-controller -o=jsonpath='{.items[0].metadata.labels.version}'

  $ kubectl get pods -n tekton-pipelines -l app=tekton-pipelines-controller -o=jsonpath='{.items[0].metadata.labels.version}'
  v0.13.2
  

[vincent-pli]

The tekton_pipelinerun_count is incorrect, that's caused by reconcile enter this block more than once:

pipeline/pkg/reconciler/pipelinerun/pipelinerun.go

Line 149 in 4fb1078

if pr.IsDone() {

[vincent-pli]

/kind bug

[RafaeLeal]

Do you think that could be the case for the duration metrics as well?
Since this block is running more than once and it calls metrics.DurationAndCount

pipeline/pkg/reconciler/pipelinerun/pipelinerun.go

Line 169 in 4fb1078

err := metrics.DurationAndCount(pr)

[vincent-pli]

@RafaeLeal
I think so, you could apply my fix and take a try.

[pritidesai]

@vincent-pli @RafaeLeal we have similar issue reported for tekton_taskrun_count, see #3029

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[tekton-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

[pritidesai]

/remove-lifecycle rotten
@RafaeLeal you still experiencing this issue?

[ibexmonj]

hey folks, I am curious if what i reported here #3739 about some of the pipeline metrics is relevant to this discussion.

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale with a justification.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[tekton-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

[tekton-robot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen with a justification.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/close

Send feedback to tektoncd/plumbing.

[tekton-robot]

@tekton-robot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen with a justification.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/close

Send feedback to tektoncd/plumbing.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.