[integrations] Tekton Dashboard

[wlynch]

Tracking issue for integrating with the Tekton Dashboard.

We'd like to surface Results/Records in the Tekton UI so that users can visualize the logic groupings / extra data Results can store. A really rough version of this can be found here: https://github.com/tektoncd/results/tree/main/tools/simpleui

I am not sure how we can do this in a minimally invasive way. Open to ideas!

[Fabian-K]

Unfortunately not an idea of how to go forward but want to say that this is really exciting. Due to relying on the Kubernetes APIs the dashboard really struggles when there are lots of resources. Implementing proper pagination is difficult and also filtering possibilities are limited. As a workaround, we try to keep the number of resources low (e.g. using https://github.com/tektoncd/experimental/tree/master/pipeline/cleanup#cleanup-old-taskruns-and-pipelineruns) but ideally we should be able to keep as much history for e.g. all executions of a pipeline as we want and not as technically limited ;)

Would be awesome if results is the solution!

[wlynch]

@Fabian-K Sorry I missed this! But yes, that's the goal for this project. :)

This issue is scoped purely for integrating with the Tekton Dashboard, but you'll probably find #59 useful to your use case!

[wlynch]

https://github.com/tektoncd/dashboard/blob/master/docs/extensions.md looks like a promising way to hook into the Dashboard.

[AlanGreene]

@wlynch I'd love to talk more about the desired user experience for viewing / interacting with the Results data. There are plenty of options to explore depending on the desired outcome.

At a high level I see 2 main options for the user:

• add new pages to the Dashboard to surface data from the Results API using the existing extensions mechanism
  • Note: Dashboard extensions were originally intended to allow for both adding new pages and customisation of existing pages. Unfortunately this plan was never fully realised and extensions have remained pretty basic, only supporting the addition of new pages to the app (either simple auto-generated pages for Kubernetes resources, or custom UI similar to the deprecated webhooks-extension).
• alternatively/additionally, provide better integration into the existing UI, so for example we might populate the existing PipelineRun details page with data from the Results API if the resources are no longer available on the cluster

There are some planned changes to the Dashboard's API / data layer that may make integration / customisation easier in future, and some other challenges such as auth that we'll need to address too.

[ghost]

• alternatively/additionally, provide better integration into the existing UI, so for example we might populate the existing PipelineRun details page with data from the Results API if the resources are no longer available on the cluster

This seems like the ideal route to me. Results providing the historical record / source of truth for anything that's reached a Completed ( / deleted) state.

@wlynch do you think that's fair assessment?

Are the dashboard's data layer plans related to the redux removal issue or documented elsewhere?

[AlanGreene]

Are the dashboard's data layer plans related to the redux removal issue or documented elsewhere?

Yes that's the first part of it, a major simplification of the data layer (and corresponding reduction in boilerplate) so we can more easily plugin additional / alternative sources while solving some of the performance and caching issues we have today. I need to flesh out the plan a bit more and document it so we can start to make progress. I have my hands full for the next 1 - 2 weeks but will try to get something on paper asap for others to review.

[wlynch]

• alternatively/additionally, provide better integration into the existing UI, so for example we might populate the existing PipelineRun details page with data from the Results API if the resources are no longer available on the cluster

This seems like the ideal route to me. Results providing the historical record / source of truth for anything that's reached a Completed ( / deleted) state.

@wlynch do you think that's fair assessment?

Yup! The Result API also captures data for unique instances of Runs with the same name (e.g. if you create->delete->create) which would probably also be useful for the dashboard.

Modeling the dashboard on the Result API would also let us display related data (e.g. PipelineRun + TaskRuns) in groups.

@AlanGreene I added a discussion item for the March 8th Dashboard WG so we can talk about questions/ideas!

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale with a justification.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[pietervincken]

Is this issue still actively being worked on?
(I came here from the Dashboard project due to the slowness of the dashboard)

[wlynch]

@AlanGreene it's been awhile since we've touched base on this. Any updates on the dashboard side for how to get started experimenting?

[tekton-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

[tekton-robot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen with a justification.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/close

Send feedback to tektoncd/plumbing.

[tekton-robot]

@tekton-robot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen with a justification.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/close

Send feedback to tektoncd/plumbing.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[tekton-robot]

@pietervincken: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

This feature is still missing. IIRC this was the endgoal of this subproject?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[pietervincken]

Can we re-open this issue? We're running Tekton on a larger scale (2k+ pipelineruns, 20k+ taskruns active in the cluster) so we'd be very interested in having the ability to not rely on the Kubernetes API for looking into results of a pipeline. Currently we need to very aggressively clean the runs in order for the dashboard to stay performant.

[wlynch]

/lifecycle frozen

This is something we still want to get to, but need help from the dashboard team 🙏

[drriguz]

Hi, guys, any progress on the integration?

Currently it's not possible to view the build status of a project via dashboard like jenkins, and it's also not directly supported through the pipeline metrics(use grafana + prometheus). So I'm wondering if this could be achived by tekton results?

[jb-2020]

alternatively/additionally, provide better integration into the existing UI, so for example we might populate the existing PipelineRun details page with data from the Results API if the resources are no longer available on the cluster

+1 to the second approach recommended by @AlanGreene - It would be great to leverage results api to make the PipelineRun link (e.g. namespaces/tekton-pipelines/pipelineruns/foo-123 permanent with minimal overhead in etcd as well as persist the logs as mentioned in: #198

These features would greatly improve the out of the box experience of Tekton IMO.

[vdemeester]

/area roadmap

[xinnjie]

/assign

[xinnjie]

@AlanGreene @adambkaplan
I'd like to propose a TEP about Results visualization in Dashboard in a few day.

The general idea is:
In PipelineRun/TaskRun list and detail view, first query Results API, if PipelineRun/TaskRun is not finished, fall back to Kubernetes API.

Benifits:

1. Resource URL in deashboard will match Results API perfectly.
2. If first query Kubernetes API and then query Results API, there will be be some pagination problems, because we need to take care of both API pagination.
3. Compactbility: During development, we could set up 2 new tabs: PipelineRun results and TaskRun results for test. Modification to oringal PipelineRun/TaskRun tab would be as less as possible(Although there would be some refactors to make 2 sides use same impletation). When this feature is stable and Results API is enabled, replace original view with results view. If Results API is not enabled, all things go like noew.

[sayan-biswas]

1. We need to modify Results Watcher reconsilation. Currently watcher not produce result until PipelineRuns/TaskRuns ended.

@xinnjie I guess watcher does create the PipelineRuns/TaskRuns entries in the database as soon as it's reconciled, irrespective of the status. If that is what you meant.

[khrm]

I created a basic dashboard based on results using Flutter.
https://github.com/khrm/bowspirit Lot of things still need to be done on it. Atm, it has a list for Pipelinerun, also view Pipelinerun's Yaml and Log

[xinnjie]

1. We need to modify Results Watcher reconsilation. Currently watcher not produce result until PipelineRuns/TaskRuns ended.

@xinnjie I guess watcher does create the PipelineRuns/TaskRuns entries in the database as soon as it's reconciled, irrespective of the status. If that is what you meant.

Yes, it is. It's my mistake here.

[cmorinupgrade]

Any progress on enabling the dashboard to fallback to Results API database?

@xinnjie did you end up implementing that TEP you were referring to? Thank you! 🙇

[AlanGreene]

@xinnjie Is this something you're still planning to work on or do you have some progress update you can share? If you're not planning to work on it that's fine, please unassign yourself so someone else can pick it up. Thanks.

[avinal]

It seems Xinnjie is not active on GitHub since October. it should be okay to unassign this issue.

[GijsvanDulmen]

This is really a missing feature in the complete Tekton ecosystem. Would easy up resources on the cluster as well.

[xinnjie]

Sorry guys, I'm not available for now.

[Stringls]

Any updates on that?
It would be really helpful to save logs of PipelineRuns in a Tekton Dashboard

[AlanGreene]

@Stringls Can you elaborate on what you're looking to do? If it's just persisting logs and loading those in the Dashboard see https://github.com/tektoncd/dashboard/blob/main/docs/walkthrough/walkthrough-logs.md. It describes a way to achieve this today, using minio for storage as an example, but can be achieved using any storage provider you wish as long as you can provide the required API for the Dashboard to consume.

If you're looking to persist the Tekton resources too, and load both those and the logs in the Dashboard from Tekton Results, that is not yet supported. There are a number of blockers on the Results side that need to resolved first before the Dashboard could add this support, including #535

[Stringls]

@AlanGreene Sure.
So as a user of Tekton I expect to:

• Run Tasks/Pipelines
• have UI (Tekton Dashboard)
• have persistent Logs of the executed Tasks/Pipelines

As we know, we lose logs when pods of the TaskRun/PipelineRun are deleted, but I would like them to be persistent and not be deleted as Logs is a crucial part of any CI.

Setting up minio as a storage of the logs sounds useful, but not ideal. IMHO there should be a native solution in Tekton Results without any further configuration of other tools (e.g. minio). For example, if a user wants to have a local storage, Tekton Results monitors every TaskRun/PipelineRun and saves logs (stdout, stderr) in PVC/s and once Tekton Pruner deletes TaskRuns/PipelineRun Tekton Results deleted the logs associated with that Tekton resources in its storage.

We don't really look to persist the Tekton resources, currently.