TEP-0100: Embedded TaskRuns and Runs Status in PipelineRuns [Proposal]

[jerop]

In TEP-0100, we described the challenges caused by embedding
the complete TaskRun and Run status in PipelineRuns: performance
degradation, memory bloat, and lack of extensibility.

In this change we add the proposal to reduce the amount of information
stored about the status of TaskRuns and Runs to only the object
references in order to improve performance, reduce memory bloat
and improve extensibility. We also mark the TEP as implementable.

Co-authored-by: Lee Bernick leebernick@google.com

/kind tep

/cc @vdemeester @imjasonh @bobcatfish @afrittoli
(approvers of the problem statement added in #606)

[bobcatfish]

/assign

[bobcatfish]

This is great!! Thanks for being so thorough

/approve

(need to get some non-google approvers on this as well)

[tekton-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bobcatfish

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• teps/OWNERS [bobcatfish]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jerop]

@tektoncd/core-maintainers we're looking for non-googler reviewers here 🙏🏾

(thanks @bobcatfish and @wlynch for the reviews)

[pritidesai]

Hey @AlanGreene please help us review this proposal! Does this solution aligns with the way we have truncated pipelineRun status and accessing it when needed?

[imjasonh]

/assign

[AlanGreene]

This will be a welcome change in terms of the performance improvements to be gained for users and other consumers. I have a few small concerns about impact on the Dashboard, but hopefully we can work around most of them relatively easily. See the comments.

One other thing to point out is the potential impact on our own use of the Dashboard in the dogfooding cluster. We currently have scheduled cleanup of resources meaning that for a given PipelineRun its TaskRuns may no longer be available on the cluster. The Dashboard falls back on data in the PipelineRun.status.taskRuns today to attempt to fill in the gaps and display most of the relevant information. We would lose this ability with the removal of that field. I don't think this is a blocker but figured it was worth mentioning so it doesn't come as a surprise later.

[afrittoli]

Thank you for this TEP!

I'm definitely in support of the idea. My main comment would be go actually go further and do not store the status of child resources at all, only type and part of the meta (name and possibly namespace/cluster in future).
Since we need to change the API to make this happen, and we need to go through the 9 months deprecation cycle, I see no reason to only remove part of the duplicated data.

I would love to see v1 only having the essential information to identify child resources and associate them to the part of the Pipeline they belong to (i.e. PipelineTask).

[dibyom]

For the duckv1beta1.Status part, I would not include it, because it means again extra space, extra writes to the PipelineRun so extra reconcile cycles triggered - and I don't see the advantage for the controller of having that information locally in the PipelineRun resource.

@afrittoli I agree that this will save space. How will this impact current API consumers? Will they have to change their logic to make extra API calls or are they doing that already? (/cc @tektoncd/cli-maintainers @tektoncd/dashboard-maintainers )

[jerop]

@afrittoli @pritidesai @dibyom @wlynch @vdemeester @AlanGreene @piyush-garg @imjasonh @bobcatfish - thanks for the reviews! updated the TEP and it's ready for another look :)

[imjasonh]

/lgtm

This looks great, thanks for pushing on this! 👍

[jerop]

Thanks everyone for the reviews! We resolved the open questions at API working group yesterday, and the TEP was updated as discussed. This TEP was merged offline, with all assigned approvers having signed off. If anyone has any more comments or concerns, please let me know 😁