taskrun counter metrics grow up without failed taskrun

[kubovdm]

Expected Behavior

tekton_pipelines_controller_taskrun_count{status="failed"} grow up only if taskrun failed and only increase by 1

Actual Behavior

I have alerts about taskrun failed without failed taskrun and then check counter in prometheus and counter is growing up.

Steps to Reproduce the Problem

Install tekton with helm and setup prometheus.

Additional Info

• Kubernetes version:

Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.4", GitCommit:"b695d79d4f967c403a96986f1750a35eb75e75f1", GitTreeState:"clean", BuildDate:"2021-11-17T15:48:33Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.10-gke.1600", GitCommit:"ef8e9f64449d73f9824ff5838cea80e21ec6c127", GitTreeState:"clean", BuildDate:"2021-09-06T09:24:20Z", GoVersion:"go1.15.15b5", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.22) and server (1.20) exceeds the supported minor version skew of +/-1

- Tekton Pipeline version:

**v0.27.1% **


<!-- Any other additional information -->
One more fing. For one minutes counter jumped too mach
1750 @1641551193.666
1764 @1641551223.673
1778 @1641551253.666

and for one taskfail i have grow counter with two
and Sorry for my english

[jerop]

thanks for the issue report @kubovdm!

here's a related issue for future reference: #3739

[wlynch]

/assign @chuangw6

[tekton-robot]

@wlynch: GitHub didn't allow me to assign the following users: chuangw6.

Note that only tektoncd members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

/assign @chuangw6

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[dibyom]

/cc @khrm might be working on this?