Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Design PipelineResource extensibility #238

Closed
bobcatfish opened this issue Nov 10, 2018 · 23 comments
Closed

Design PipelineResource extensibility #238

bobcatfish opened this issue Nov 10, 2018 · 23 comments
Assignees
Labels
design This task is about creating and discussing a design

Comments

@bobcatfish
Copy link
Collaborator

Expected Behavior

We should have clear extension points in the Pipeline CRD system. These points should be well documented, clear, and tested.

Ideally it will be possible to extend the system without having to modify the existing controller binaries or CRD definitions.

Actual Behavior

At the moment, if a user wanted to extend the Pipeline they have these options:

  1. They want to create their own kind of Resource (e.g. they use something like mercurial, but we currently only support git) = They have to modify and re-deploy the TaskRun controller @_@ (and maybe the PipelineRun controller also)
  2. They want to create their own kind of Task = Design alternative Task Implementations inside a pipeline #215 + ducktyping should allow for alternative Task implementation
  3. They want to run xyz specific logic = They can put any logic they want inside a task

In the worst case, the last option should allow a user to do pretty much anything they want, outside of the functionality provided by this system.

Additional Info

n/a

@bobcatfish bobcatfish added the design This task is about creating and discussing a design label Nov 10, 2018
@bobcatfish bobcatfish added this to the 0.0.1 Alpha release milestone Nov 10, 2018
@bobcatfish bobcatfish self-assigned this Nov 10, 2018
@bobcatfish
Copy link
Collaborator Author

Here is one proposal, which I am unreasonably attached to:

Similar to how consourse uses Resources as an extension point, we could also use Resources as an extension point.

  1. We add a Resource controller, which only pays attention to Resource types it knows about and ignores the rest
  2. When the Resource controller sees a Resource it knows about, it will make that Resource available via a PVC (e.g. if it's a Git resource check out the repo into a volume mount) or via some string data in the Status of the resource (e.g. for an image we just need the url + digest)
  3. If a user wants to add their own kind of Resource, they can add a controller that knows how to handle that Resource type
  4. When the Resource is used as an output, maybe this updated via some fields in the spec, which the controller can react to?

Cons:

@abayer
Copy link
Contributor

abayer commented Nov 12, 2018

Seems reasonable at first glance - I'm putting together some thoughts on extensibility based on some design work here (i.e., we're establishing what we want to be able to accomplish to do something like automatic quality gates on Tasks), and will try to get that written up to figure out how that would fit in with this approach.

@nader-ziada
Copy link
Member

We will probably need to have a PVC anyways to share outputs from one taskRun (in a pod) with another taskRun in the same pipeline (in a different pod)

@cppforlife
Copy link

This requires the system we deploy to support PVCs

last time i was playing with azure's kubernetes service attaching volume to a node was taking ~2min whereas on gke it was only 10s. i think if persistent volumes are used then we should figure out how to deal with long attachment times (or how to avoid them entirely if that's possible).

@tejal29
Copy link
Contributor

tejal29 commented Nov 14, 2018

@pivotal-nader-ziada would downloading the output from a result store be a viable option ?

@nader-ziada
Copy link
Member

@tejal29 you mean something like gcs or s3 for sharing artifacts between tasks? yes, this would be an option, but its one extra thing users have to setup and have ready. Based on @cppforlife comment and my own investigations, I want to avoid using PVC, but still investigating best options to discuss with everyone

@abayer
Copy link
Contributor

abayer commented Nov 14, 2018

Something worth discussing - what exactly do we want the extensions to be able to do? Some things definitely map directly to Resources - different SCM systems, different artifact storage, etc. But is that actually the only thing we want to be able to extend? For example, there's notifications (i.e., #49) - obviously, we can't hardcode in every notification system someone might want to use, so we need there to be a way to have extensions provide additional notification systems. But are those Resources?

@nader-ziada
Copy link
Member

I think of resources as the input and outputs of the task, so notifications can be expressed as output of the task. If we design the extension point with enough flexibility, the resource should be responsible for taking the result of the task and doing what it wants with it including sending it to slack for example.

@abayer
Copy link
Contributor

abayer commented Nov 15, 2018

Ok, so in cases like notifications, we wouldn't be using PVC, right?

@nader-ziada
Copy link
Member

Yes, we would not be using PVC for the notifications, we are trying to minimize any use of PVC in general for performance reasons

@abayer
Copy link
Contributor

abayer commented Nov 15, 2018

It would be helpful to lay out some more examples of what we want to be extensible in the first place - notification mechanisms and SCM checkout processes are pretty clear to me, but I'm not sure what else is being considered here?

@bobcatfish
Copy link
Collaborator Author

last time i was playing with azure's kubernetes service attaching volume to a node was taking ~2min whereas on gke it was only 10s. i think if persistent volumes are used then we should figure out how to deal with long attachment times (or how to avoid them entirely if that's possible).

Great point @cppforlife , thanks for sharing those stats!

Something worth discussing - what exactly do we want the extensions to be able to do?

@abayer that's a great point - we should try to be clear about what we want users to be able to extend. It's hard to know in advance though :S

Definitely we want them to be able to provide their own:

  • Places to get data from (e.g. images, version control systems)
  • Places to put data (e.g. registries, repositories)

(Maybe what I'm doing here is defining what a Resource is ... lol as it slowly becomes a Concourse resource minus polling...)

I created this doc for brainstorming - knative-dev@ should have edit access.

For example, there's notifications (i.e., #49) - obviously, we can't hardcode in every notification system someone might want to use, so we need there to be a way to have extensions provide additional notification systems. But are those Resources?

Depends on how we design notifications! They could be viewed as a type of resource (something you put data to) like @pivotal-nader-ziada described or we could make them a top level thing in the system, e.g. we have Resources, Tasks, Pipelines + Notifications. I can see either working well.

@bobcatfish
Copy link
Collaborator Author

Another note re. using PVCs, I realized the design I proposed in #238 (comment) has a big flaw: if multiple Pipelines, or even Tasks within a Pipeline, try to use the same Resources as inputs, they'd be mounting in the same PVC, which seems like it either wouldn't work or would be asking for some terrible side effects.

Here's an updated proposal (specifically for extending Resources) which I think is better:

  1. We add a Resource controller, which only pays attention to Resource types it knows about and ignores the rest
  2. When the Resource controller sees a Resource it knows about, it will make that Resource available via a PVC (e.g. if it's a Git resource check out the repo into a volume mount) create the spec for a corev1.Container (i.e. the same way that build specifies steps) that knows how to make that Resource available in a location expected by the Task (maybe to the workspace path?) in the Resource's status section
  3. If a user wants to add their own kind of Resource, they can add a controller that knows how to handle that Resource type
  4. When the Resource is used as an output, maybe this updated via some fields in the spec, which the controller can react to?

@bobcatfish bobcatfish removed this from the 0.0.1 Alpha release milestone Nov 20, 2018
@bobcatfish
Copy link
Collaborator Author

Relevant comment from @mattmoor in slack:

I am wondering it PipelineResource would benefit from being a duck type instead of a concrete type...
e.g. right now you bake in type:, which isn't particularly extensible: https://github.com/knative/build-pipeline/blob/0d6a3a27a05ba1a765b6f28a434a2c76baf883ca/examples/pipelines/kritis-resources.yaml#L7

@bobcatfish
Copy link
Collaborator Author

Okay so I'm feeling pretty good about this plan, what do folks think:

For both PipelineResources and Tasks we allow extensiblilty by using duck typing, i.e. defining an interface which they must each comply with.

  1. For PipelineResources, we'll define controllers that know what types of Resources they can handle.
  2. For Tasks, we already provide a generic TaskRun controller, the key will be to update PipelineRuns so that they can know how to create other Run types for other Task types (e.g. was talking with @balopat about having a SkaffoldTask)

bobcatfish added a commit to bobcatfish/pipeline that referenced this issue Jan 4, 2019
We use the syntax `somethingRef` in several fields on our specs because
we want to be able to use ducktyping down the line to switch out other
types (see tektoncd#238).

However `triggerRef` is a bit weird because:
- If the type is `manual`, there is no actual CRD to reference
- We don't know what other sorts of types we might need (e.g. when
  created via events)
- It was inconsistent b/w TaskRun and PipelineRun (TaskRun had an extra
  level of indirection)

This commit applies @sebgoa's suggestion to simplify this to just
`trigger`.

This was part of the discussion in tektoncd#320 about simplifying the
interfaace.
knative-prow-robot pushed a commit that referenced this issue Jan 4, 2019
We use the syntax `somethingRef` in several fields on our specs because
we want to be able to use ducktyping down the line to switch out other
types (see #238).

However `triggerRef` is a bit weird because:
- If the type is `manual`, there is no actual CRD to reference
- We don't know what other sorts of types we might need (e.g. when
  created via events)
- It was inconsistent b/w TaskRun and PipelineRun (TaskRun had an extra
  level of indirection)

This commit applies @sebgoa's suggestion to simplify this to just
`trigger`.

This was part of the discussion in #320 about simplifying the
interfaace.
@bobcatfish
Copy link
Collaborator Author

Note to self in #414 we are adding rules around which types of resources produce outputs - something to take into account in a design where resources are extensible :D

@hrishin
Copy link
Member

hrishin commented Apr 29, 2019

Hi @bobcatfish 👋

We would like to know, how this issue is coming along? have we started working on controller implementation for PipelineResource or we are expecting this design to evolve more?

When the Resource is used as an output, maybe this updated via some fields in the spec, which the controller can react to?

Also, would you mind elaborating this point? 🤔

@bobcatfish
Copy link
Collaborator Author

We would like to know, how this issue is coming along? have we started working on controller implementation for PipelineResource or we are expecting this design to evolve more?

No implementation yet! I'm going to try to flesh out the design a bit further asap :D

@bobcatfish
Copy link
Collaborator Author

Design doc focusing on PipelineResource based extensibility available at: https://docs.google.com/document/d/12SPwIHZpERbFGroQulu3A16Sp-sPODAGP8b78-3WQDs

(I realized I actually don't have a clear idea of how to design custom Tasks, so if anyone wants to take on that design work, that's still available!)

@bobcatfish bobcatfish changed the title Design Pipeline extensibility Design PipelineResource extensibility May 8, 2019
@bobcatfish
Copy link
Collaborator Author

I'm gonna downscope this issue to be just about PipelineResource extensibility, and we can use #215 for Task extensibility!

@AdityaGupta1
Copy link

Relevant discussion here: https://tektoncd.slack.com/archives/CJ62C1555/p1560974129126900

It would be nice if there was a way to specify a file/directory or even just a simple text value to be passed between tasks without needing to create a volume across tasks. Maybe even a non-typed PipelineResource?

tekton-robot pushed a commit to tektoncd/catalog that referenced this issue Jun 26, 2019
This task is used to sync an Argo CD application with its underlying Git repository and to wait for it to become healthy. It takes in no resources, only parameters, and passes them to the argocd command. The application's name and the Argo CD server address are required. Some form of login is also required, either through a username/password combo or an authentication token. The specific revision to sync to and the flags to append to commands can also be passed as parameters.

Currently there is no way to pass plain strings or files between tasks, or it could have been better to split the login and sync tasks. This way, there could have been two login tasks - one for username/password and one for authentication token. With the current approach, the bash command runs an if statement to see if the token's default value was overriden, and if so, it uses that token. Otherwise, it uses the username/password passed in. This issue (typing of PipelineResources) is being discussed in tektoncd/pipeline#238.
@bobcatfish bobcatfish assigned vdemeester and unassigned bobcatfish Aug 5, 2019
@bobcatfish
Copy link
Collaborator Author

@vdemeester @pmorie what are the next steps for the design here? I think folks are pretty happy with the design as proposed and we could close this issue, and move into implementing it, what do you think?

@bobcatfish
Copy link
Collaborator Author

I'm gonna close this! @vdemeester @pmorie can we make some follow up issues to actually implement this? Happy to help if needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
design This task is about creating and discussing a design
Projects
None yet
Development

No branches or pull requests

8 participants