Dashboard hangs when pipelineruns are in 1000's

[BhawaniSingh]

We have been evaluating tekton in testing for over 9 months. We are confident that we could use it as a primary CI/CD system instead of Jenkins. But we are facing a problem with tekton (tekton dashboard).
Although the tekton controller can handle thousands of jobs per day, the dashboard becomes unresponsive or takes too long to load when there are more than a couple of thousand pipelineRuns. This is currently our primary issue.

We evaluated tekton results, it surely has a great future, but unfortunately, it lacks a usable dashboard and is not integrated with the tekton dashboard.

To address these challenges, I would like to have your insights on the following:

1. What options do we have to provide better user experience apart from tekton dashboard? Are there any other alternatives?
2. Are there any plans to integrate tekton results with the tekton dashboard?
3. We are currently running tekton pipelines and the tekton dashboard in their default configurations. Are there any recommendations you could provide to improve tekton dashboard's performance?

Please let me know if there's anything I can do to help? (Please do note that I have very little knowledge of the go programming language)

[AlanGreene]

Hi @BhawaniSingh, thanks for raising an issue. Please see this previous response to a similar issue: #1978 (comment)

In general, using the Kubernetes cluster / etcd for long term storage of your CI/CD runs in this manner is not advised. As the number of resources grows performance will decrease, affecting not just the Tekton Dashboard but all clients accessing the cluster, including kubectl etc. Also bear in mind that keeping PipelineRuns may mean that they're holding onto volume claims and other system resources, tying up storage and other resources that could otherwise be used.

Tekton runs much of its own CI/CD on Tekton, and as mentioned in the link we have automation to periodically remove old runs from the cluster to keep the numbers reasonable.

Integration with Tekton Results is on the roadmap, link in the comment above.