[TEP-0044] Update Alternatives

[lbernick]

This commit describes design constraints related to running multiple Tasks in one pod in more detail.
It also updates the alternative solutions based on the clarified problem statement.

Changes:

• Add 3 options: pipeline in a pod + pipelines in pipelines, pipeline -> taskrun, and task pre and post steps
• Modify and add more detail to the "TaskGroup" proposal
• Remove "focus on workspaces" option: this adds new syntax but not new functionality
• Remove "custom pipeline" solution due to similarity with grouping CRD solution
• Remove "custom scheduler" option: This is a solution for scheduling parallel Tasks that share a PVC
  on the same node, but does not address the problems of pod overhead and the need
  to share data between tasks in the first place.

Co-authored-by: Christie Wilson christiewilson@google.com @bobcatfish

/kind tep
/assign vdemeester
/assign afrittoli

[dibyom]

/assign @dibyom
/assign @jerop

[lbernick]

@dibyom @jerop @afrittoli @vdemeester have you had the chance to look at this yet? This is just updating alternatives, not proposing one of them, so I think this can be merged if the pros/cons adequately reflect each alternative. I'd like to be able to move forward with proposing a design.
Would it be helpful for me to separate the "design consideration" section into a separate PR?

[bobcatfish]

@lbernick drive by comment (and good chance we already discussed this so my apologies!!) but is it possible to add a bit more detail to "Remove "custom scheduler" option: this is a solution to a different problem" - it doesnt address the multiple pod issue but i think it is a way of trying to address the data sharing option in that at least if you can schedule pods that access a volume on the same node, they get faster access to that volume

(i dont think this is a solution we should go with because i dont think it addresses all of the requirements well but im not seeing it as addressing a totally different problem?)

[lbernick]

@lbernick drive by comment (and good chance we already discussed this so my apologies!!) but is it possible to add a bit more detail to "Remove "custom scheduler" option: this is a solution to a different problem" - it doesnt address the multiple pod issue but i think it is a way of trying to address the data sharing option in that at least if you can schedule pods that access a volume on the same node, they get faster access to that volume

Thanks for the feedback! I added some more detail to the commit message to elaborate. If you'd prefer I keep this alternative in there because there is overlap between what that is solving and the scope of this TEP, I'm happy to add it back.

[lbernick]

/hold

needs some updates in response to #610

[bobcatfish]

I added some more detail to the commit message to elaborate. If you'd prefer I keep this alternative in there because there is overlap between what that is solving and the scope of this TEP, I'm happy to add it back.

Thanks for updating the commit message - i think I see what you're saying that it does make it faster to share a PVC but doesn't remove the need for one. sgtm to keep it out of scope!

[lbernick]

/hold cancel

Rebased onto #610. The main changes I've made as a result are 1. rewriting the "task grouping" option so that the desired syntax can be specified on either the pipeline or pipelinerun and 2. Making the pros/cons of each alternative more clear with regards to whether they allow specifying runtime and/or authoring time user configuration.

[dibyom]

The proposed options make sense. Just had one question around specifying this in a pipeline vs a run

[dibyom]

I'm missing some background here - what's the use case for modifying the execution config in a PipelineRun vs a Pipeline?

[lbernick]

it's very likely we will want a configuration option here on the Pipeline; I think this is probably something we should implement. The reason to allow people to specify this on a PipelineRun is generally because the choice of how to execute a Pipeline is a runtime level concern. Specifically, I can imagine situations where you might have different approaches to PVCs in different clusters but want to reuse the same Pipeline definition from a repo, or maybe some PipelineRuns will use different amounts of data than other PipelineRuns for the same Pipeline and you want to sometimes distribute the work to different pods and sometimes not. Basically-- I think we shouldn't lock ourselves into a solution that can only be specified at one of runtime or authoring time. (Perhaps this should read "must be modifiable at both runtime and authoring time, but the initial implementation will be runtime based"? This is the question I tried to address with #610.)

[dibyom]

That helps, thanks. I agree that we shouldn't lock ourselves into a solution that cannot be extended.
The reason I was asking about having the config in a pipeline is that pipeline resources are specified in a pipeline/task spec. So, for the specific use case of replacing pipeline resources though (e.g. a git resource in a TaskRun), the decision of using a single pod/no PVCs was made by the task/pipeline author.
It seems like the scope of this TEP is greater than just replacing the pipeline resources use case so it might make sense to do this at the Run

[afrittoli]

Thanks for all the updates.
There may be more pros and cons one some of the options, but this looks really good.

Once this is merged, we could start building a table with various features against multiple options to help us visualise this better.

/approve

[afrittoli]

Nice. This is not addressed by finally today because using finally would require the data to be on a PVC, right?

[lbernick]

I'm not completely sure what you mean by this. This requirement is currently addressed in pipelines with finally, but the required use of a PVC causes us to run into the problems described in the problem statement. If we chose a solution where you could run the pipeline's tasks in a pod but not its finally tasks, that wouldn't address this requirement also due to the required PVC.

[jerop]

@lbernick - I think what @afrittoli is referring to is that Results from failed Tasks are not available to any other Tasks, so the test "results" have to be in a PVC (they are not Results as defined in Tekton Pipelines) - see related issue in tektoncd/pipeline#3749

[lbernick]

oh I see, thanks!

[afrittoli]

I'm guessing this is requirement for cases like: clone / hermetic build / push where the first and last need network access, the middle one doesn't?

[lbernick]

yup! there's more detail on use cases for different levels of hermeticity in a pipeline in TEP 25.

[afrittoli]

If the reason for separate pods is security, using different service accounts would force separate pods anyways.
If the reason for separate pods is resources, having high resource requirements could enforce sequential execution of the use of separate Pods.
Alternatively this could be overcome by some kind of "anti-affinity" between PipelineTasks.
This gets scarily close to the scheduling domain though - which means there may be more complexity behind this than it appears on the surface.

[lbernick]

That's a great point. I think the reason is resources, not security, and so choosing this option would essentially be trading one resource-related problem for another.

[lbernick]

FYI @dibyom @afrittoli @vdemeester since you've already approved: I decided to remove the ability to specify different levels of hermeticity as a requirement for this TEP. I looked into it a little more and it's not clear that it's feasible or that users would want it. We can continue to explore this but I'm not ready to make it a requirement yet, and want to leave out of scope for this PR.

[dibyom]

the hermetic mode might be needed for chains/hermekton. @wlynch do you think its sufficient to have hermeticity only specified on tasks running in separate pods (at least to start?)

[jerop]

looks great, thanks @lbernick!

[tekton-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: afrittoli, dibyom, jerop, vdemeester

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• teps/OWNERS [afrittoli,dibyom,jerop,vdemeester]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[wlynch]

the hermetic mode might be needed for chains/hermekton. @wlynch do you think its sufficient to have hermeticity only specified on tasks running in separate pods (at least to start?)

Still need to dig into this, but at a glance it seems odd to me if we promise different capabilities for the same type depending on the scheduling mode. I think we'll want a gameplan for how to support hermetic builds for Tasks scheduled on the same pod, even if it doesn't land right away.

@mattmoor might have some ideas here, since there's probably a fair amount of overlap with hermetic steps.

[lbernick]

the hermetic mode might be needed for chains/hermekton. @wlynch do you think its sufficient to have hermeticity only specified on tasks running in separate pods (at least to start?)

Still need to dig into this, but at a glance it seems odd to me if we promise different capabilities for the same type depending on the scheduling mode. I think we'll want a gameplan for how to support hermetic builds for Tasks scheduled on the same pod, even if it doesn't land right away.

@mattmoor might have some ideas here, since there's probably a fair amount of overlap with hermetic steps.

I'm hoping to keep hermekton discussion out of scope for this PR; once it's merged I'll open a new one specifically to discuss hermekton and loop in everyone on this thread!

[dibyom]

/lgtm