Idea: Tekton Generators

[wlynch]

/kind feature
/kind design

Creating in pipelines, since I think this is more of a discussion on our approach to Pipeline and Task configuration.

Problem

Currently it's not an easy for users to bootstrap common workflows. e.g. "run a simple build and test workflow on this GitHub repo".

Feedback I'm looking for

• Is this a good/useful idea?
• How else (if at all) can we encourage users to use and contribute catalog tasks?
  • Should we enforce any type of schema on generator configuration?
• Link to any prior art of DSLs / alternative Tekton configurations you're aware of.
• Any other ideas / improvements / critiques

Ideas

Goals:

• Provide a DSL spec for pipeline generators to automate pipeline set up for users.
• Don't make Tekton directly aware of integration types (e.g. GitHub)
• Make it easy for users to make their own.
• Encourage users to reuse existing Tekton concepts.

Simplified spec for common workflows

Ideally, it should be very easy for users to get started with only their repo and build steps. e.g. imagine a world where you configure just the following:

kind: GitHub
metadata:
  name: github-build
spec:
  url: "https://github.com/wlynch/test"
  steps:
    - name: build
      image: gcr.io/kaniko-project/executor:latest
      command:
        - /kaniko/executor
      args:
        - --context=dir://$(workspaces.input.path)/src
        - --destination=gcr.io/wlynch-test/kaniko-test
        - --verbosity=debug

(Note: this looks CRD-y, since this is how the idea started out, but I switched to a CLI approach based on feedback from Scott and other Googlers. API Version / Kind / Metadata still seem like useful identifiers, so I left this in.)

A CLI tool (likely tkn, but doesn't need to be) should be able to take this spec and when applied, should generate:

• Wrap steps in a Task
• Wrap Task in Pipeline
• Wire up workspaces for Git repo.
• Attach credentials (if necessary)
• Prepend Git clone Task at the start of the Pipeline
• Add GitHub status task to end of Pipeline / finally
• Create Trigger / EventListener for GitHub events to kick off pipeline on pushes.

All resources created should be labeled with the generator that created it, to facilitate bulk queries / deletion / modification.

Users should be able to specify the runtime configuration however they want. This means generators should support users providing one of:

• steps
• taskSpec
• taskRef
• pipelineSpec
• pipelineRef

Extensibility

To allow users to provide their own generators, I think the easiest approach would be to follow the model of Git credential helpers.

tl;dr of this approach is when given a configuration, we should look for a binary on the user's path matching tekton-generator-<generator kind>, i.e. tekton-generator-github. The generator configuration is piped in to the binary via STDIN, and the generated response is provided from STDOUT.

This allows users to define their own generators, or even invoke the generator directly to expose the underlying pipeline to modify the contents.

Ease of use

We should probably have some utilities to help interacting with generators. Some that come to mind:

• show: Print generated configuration.
• write: Write generated configuration to disk.
• apply: Apply generated configuration to Kubernetes (based on local k8s context).
• delete: Delete generated resources.

Example

I've been hacking on a really rough concept of this at https://github.com/wlynch/tekton-demo/tree/master/bundle.

[vdemeester]

Some links:

• RFC: Tekton DSL based on Starlark experimental#185
• https://github.com/redhat-developer/build/
• https://github.com/bigkevmcd/tekton-ci

[wlynch]

We're going to go ahead and start working on this in experimental.

Milestones we're going to focus on:

1. Create tool that given an input spec with steps, generates the resulting Tekton resources for the particular type.
   • Create binary that invokes another binary on the path based on the type.
   • Read input steps and generate resulting pipeline config that mounts GitHub workspace and configures steps accordingly.
2. Add support for writing output to disk.
3. Add support for tasks and pipelines
4. Add support for applying config to cluster.
5. Add support for deleting configs from cluster.
6. Add support for other types (e.g. GitLab?)

I suspect this may have overlap with the Custom Tasks proposal in the future. :)

/cc @YolandaDu1997

[YolandaDu1997]

I have created a CLI tool for the initial GitHub type of Tekton Generators.
Here are a couple of features that I’ve accomplished:
Design:

1. Designed the input config data structure and the way to parse the yaml file to store the result in the struct.
2. Designed and Built the CLI tool to help interact with generators.

Implementation:

1. Created the initial schema of GitHub generator that, given an input spec with steps, generates the resulting Tekton resources for the GitHub type.
2. Added support for writing output to disk for users to reuse these Tekton resources.
3. Added support for generating complete tasks and pipelines.
4. Created a client with kubeconfig file and added support for managing configs on the Kuebenetes cluster based on the context information of the cluster.
5. Created PipelineRun for the generated pipeline to execute the tasks.
6. Created Trigger for GitHub events to kick off the pipeline on pushes and pull requests.

Results:
With the correct input config, generated the Tekton resources and successfully ran with PipelineRun or Trigger on the Kubernetes cluster.

The next step of Tekton generators would be the #extensibility. It needs to allows users to define their own generators, or even invoke the generator directly to expose the underlying pipeline to modify the contents.

[dbazhal]

Just to add our two cents.
We're solving problem of generation tekton pipelines with tekton task. It accepts unit name, code skeleton, git branch as input params, and code(mix of python/bash/whatever) inside task creates all required for this pipeline - copies additional pipelines from other namespaces, uses some templating to dynamically create pipeline for service, creates all required triggers(trigger objects are also stored as templates inside this "generator" and are parametrized with unit name).
So when you need to create new service - you just run this generator clustertask inside your namespace with required params, and you're done. Unlimited flexibility and power :)
Additional thing we had to do - write a lightweight operator that can re-run generators in response to some events in cluster that might affect how pipelines are generated.

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale with a justification.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[tekton-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

[afrittoli]

Just to add our two cents.
We're solving problem of generation tekton pipelines with tekton task. It accepts unit name, code skeleton, git branch as input params, and code(mix of python/bash/whatever) inside task creates all required for this pipeline - copies additional pipelines from other namespaces, uses some templating to dynamically create pipeline for service, creates all required triggers(trigger objects are also stored as templates inside this "generator" and are parametrized with unit name).
So when you need to create new service - you just run this generator clustertask inside your namespace with required params, and you're done. Unlimited flexibility and power :)
Additional thing we had to do - write a lightweight operator that can re-run generators in response to some events in cluster that might affect how pipelines are generated.

Thanks @dbazhal - I discovered your comment only now. Is the work you've been doing something that you'd be willing to share?

[tekton-robot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen with a justification.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/close

Send feedback to tektoncd/plumbing.

[tekton-robot]

@tekton-robot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen with a justification.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/close

Send feedback to tektoncd/plumbing.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.