Metrics: pipelinerun_count is not correct

[vincent-pli]

Part fix issue: #2844
In scenario: PipelineResult is not null, the Metrics: pipelinerun_count will not correct.
The root cause is that:

pipeline/pkg/reconciler/pipelinerun/pipelinerun.go

Lines 149 to 154 in 4fb1078

	if pr.IsDone() {
	// We may be reading a version of the object that was stored at an older version
	// and may not have had all of the assumed default specified.
	pr.SetDefaults(contexts.WithUpgradeViaDefaulting(ctx))
	c.updatePipelineResults(ctx, pr)

The updatePipelineResults(ctx, pr) will update pr.Status.PipelineResults if needed, this will make the knative/pkg update the Pipelinerun.Status and reconcile will be trigger again.

So the pipelinerun_count will count again.

Changes

Submitter Checklist

These are the criteria that every PR should meet, please check them off as you
review them:

• Includes tests (if functionality changed/added)
• Includes docs (if user facing)
• Commit messages follow commit message best practices
• Release notes block has been filled in or deleted (only if no user facing changes)

See the contribution guide for more details.

Double check this list of stuff that's easy to miss:

• If you are adding a new binary/image to the cmd dir, please update
  the release Task to build and release this image.

Reviewer Notes

If API changes are included, additive changes must be approved by at least two OWNERS and backwards incompatible changes must be approved by more than 50% of the OWNERS, and they must first be added in a backwards compatible way.

[tekton-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign dibyom
You can assign the PR to them by writing /assign @dibyom in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[tekton-robot]

This PR cannot be merged: expecting exactly one kind/ label

Available kind/ labels are:

kind/bug: Categorizes issue or PR as related to a bug.
kind/flake: Categorizes issue or PR as related to a flakey test
kind/cleanup: Categorizes issue or PR as related to cleaning up code, process, or technical debt.
kind/design: Categorizes issue or PR as related to design.
kind/documentation: Categorizes issue or PR as related to documentation.
kind/feature: Categorizes issue or PR as related to a new feature.
kind/misc: Categorizes issue or PR as a miscellaneuous one.

[tekton-robot]

This PR cannot be merged: expecting exactly one kind/ label

Available kind/ labels are:

kind/bug: Categorizes issue or PR as related to a bug.
kind/flake: Categorizes issue or PR as related to a flakey test
kind/cleanup: Categorizes issue or PR as related to cleaning up code, process, or technical debt.
kind/design: Categorizes issue or PR as related to design.
kind/documentation: Categorizes issue or PR as related to documentation.
kind/feature: Categorizes issue or PR as related to a new feature.
kind/misc: Categorizes issue or PR as a miscellaneuous one.

[tekton-robot]

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/reconciler/pipelinerun/pipelinerun.go	84.2%	84.3%	0.1

[vincent-pli]

/kind bug

[vdemeester]

/cc @afrittoli @mattmoor

[vincent-pli]

/test pull-tekton-pipeline-integration-tests

[vdemeester]

/cc @hrishin

[hrishin]

@vincent-pli thank you for the fix!
Let's move this complete logic into different place and consolidate all metrics reporting in one place. Wonders its better to decouple this logic from the reconciler? 🤔

Although in this case count and duration may need additional care. Like not to repeat the count and consolidate duration.

[afrittoli]

I don't think this is enough to prevent false metrics from being reported.
Not knowing much about opencensus, my expectation reading this code is that DurationAndCount was idempotent, and it would take care of not submitting twice.
Now, if the controller is restarted for instance, all metrics will be submitted again.

A way to solve this could perhaps be to write metrics where the completion time is set?

[vincent-pli]

@hrishin thanks for comments, that's great if we can prevent repeat in DurationAndCount, but I guess it's difficult. as @afrittoli mentioned the DurationAndCount should be idempotent so the invoker should take care of it:

• when to call it (in this case, count the duration of pipelinerun, so should invoke when pipelinerun complete)
• prevent repeat
• maybe others

@afrittoli
completion time is not work, it already existed everytime when access
if pr.IsDone() {

About "controller restart", I think that's ok, all the metrics will be report again but with different time series.

[hrishin]

completion time is not work, it already existed everytime when access
if pr.IsDone() {
About "controller restart", I think that's ok, all the metrics will be report again but with different time series.

@vincent-pli @afrittoli, yes this fix looks promising. Just wonders how metrics aggregators behave in case of deployment restarts? We shall test this?

[vincent-pli]

@hrishin
When controller restart, all metrics will be reload again unless you delete pipelinerun or other stuff at the restart moment.
For example: for pipelinerun we has metrics:

• pipelinerun_duration_seconds
• pipelinerun_count
• running_pipelineruns_count

All the metric list above will be recover just one thing: the pipelinerun_count become correct, since if pr.IsDone() { access only onece after controller restart : )

[afrittoli]

I meant to submit the metric in a different place in the code, when the completion time is set, which happens only once, regardless of further reconciles or restarts.

Could we add test coverage for this?

[vincent-pli]

@afrittoli
The point completion time is set is only happen once, that's right, but at the point the reconcile is not complete since the final "status: success" update is not occurred, basically, it's not a real complete pippelinerun at that point, I think we could not launch metric to mark it's already complete.

pipeline/pkg/reconciler/pipelinerun/pipelinerun.go

Lines 465 to 466 in 08ac72c

	case corev1.ConditionTrue:
	pr.Status.MarkSucceeded(after.Reason, after.Message)

[tekton-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

[tekton-robot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Send feedback to tektoncd/plumbing.

[tekton-robot]

@tekton-robot: Closed this PR.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Send feedback to tektoncd/plumbing.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.