0030-workspace-paths.md

title	authors	creation-date	last-updated	status
workspace-paths	@sbwsg	2020-10-07	2020-10-18	proposed

TEP-0030: Workspace Paths

• Summary
• Motivation
  • Motivation 1: Declaring Workspace Content
  • Motivation 2: Improving Tekton's Runtime Validation of Workspaces
  • Motivation 3: Catalog Tasks Already Do This
  • Motivation 4: "Woolly" Reasons
  • Goals
  • Non-Goals
• Requirements
  • User Stories
    • Clearly declare credential requirements for a git Task
    • Validate that webpack.config.js Build configuration is passed
    • Enable arbitrary paths to contain package main
    • Team writing TaskRuns has specific requirements for directory structure of config-as-code
• References
  • Related Issue
  • Related Designs
  • Catalog Examples

Summary

Motivation

This TEP is motivated by several ideas:

• Tasks should be able to declare the content they expect to receive via a Workspace as well as the content they produce on to a Workspace.
• Pipelines should be able to declare dependencies on specific files or paths from one PipelineTask to be used by another PipelineTask.
• TaskRuns and PipelineRuns should be able to declare when a path a Task/Pipeline is expecting lives at a different location on the Workspace that the Run is binding. They should also be able to override the location that a Task/Pipeline will write content to on a Workspace.
• Tekton should be better able to validate workspaces at runtime.
• Catalog tasks are already doing this in an ad-hoc way and support should be made consistently available.

There are also some slightly more "woolly" reasons which I'll summarise at the end of this section.

Motivation 1: Declaring Workspace Content

Tasks

Task authors currently express whether a workspace is optional as well as where that workspace will be mounted. Workspace Paths adds an option for Task authors to declare important files or directories that must either be provided on a received Workspace or created on a Workspace during execution.

Example:

A git Task exposes an ssh-credentials workspace that expects the following files: private-key (with default path "id_rsa") and known_hosts (default path "known_hosts").

The same git Task exposes an output workspace that lets the TaskRun choose a subdirectory of the Workspace to clone a repo into.

Pipelines

Pipeline authors currently express which workspaces are given to which PipelineTasks. Workspace Paths allows Pipeline authors to define files that a PipelineTask should produce that other PipelineTasks will consume.

Example:

A git-clone PipelineTask checks out a repo and the Pipeline declares that this checkout should include a Dockerfile. The Dockerfile is then fed to the docker-build Task. The relationship between the two PipelineTasks should be understood as a runAfter relationship: git-clone must run first and docker-build should run second.

TaskRuns and PipelineRuns

TaskRun and PipelineRun authors can currently bind specific volume types to Workspaces. They should additionally be able to override the expected location of important files and directories on a Workspace.

Example:

A git Task expects a Workspace with a private-key path on it, with default location of "/id_rsa". A TaskRun can override it to look instead at "/my-keys/id_ecdsa".

Motivation 2: Improving Tekton's Runtime Validation of Workspaces

The only validation Tekton currently performs of Workspaces is against the Workspace structs in the controller. The Workspace Paths feature would allow for more precise runtime validation of the paths a workspace has provided or left out.

Example: In a node-webpack-build Task, a missing webpack.config.js file could result in a uniform error before the Task's Steps are allowed to run: Workspace path missing: workspace "source-repo" does not provide "webpack_config": expected path "/workspace/source-repo/webpack.config.js".. This would put the TaskRun (or PipelineRun) into a clear failed state with the exact Task that expected the file along with which file was missing.

Motivation 3: Catalog Tasks Already Do This

There are several patterns becoming more common in the catalog now that workspaces are seeing more use:

1. Tasks declare both a workspace as well as a param for a path into the workspace. Here's an example from our catalog:

   workspaces:
   - name: source
     description: A workspace where files will be uploaded from.
   - name: credentials
     description: A secret with a service account key to use as GOOGLE_APPLICATION_CREDENTIALS.
   params:
   - name: path
     description: The path to files or directories relative to the source workspace that you'd like to upload.
     type: string
   - name: serviceAccountPath
     description: The path inside the credentials workspace to the GOOGLE_APPLICATION_CREDENTIALS key file.
     type: string

   Note above that the path and serviceAccountPath params are providing information specific to the source and credentials workspaces. Two things fall out from this:

   • Catalog Task authors clearly see a need to allow the paths into Workspaces to be customisable by users of their Tasks.
   • The more workspaces that are needed for a Task, the more params the author needs to include to provide this customisable path behavior.

   At time of writing there are 30 catalog entries employing this pattern. See the Catalog Examples reference for links to all of those.

2. Workspace descriptions are required to explain both the purpose of the workspace and the files that are expected to be received on them. Example from recent tkn PR:

   • kubeconfig: An optional workspace that allows you to provide a .kube/config file for tkn to access the cluster. The file should be placed at the root of the Workspace with name kubeconfig.

Motivation 4: "Woolly" Reasons

The following items also motivate this TEP but to a lesser degree:

• Decouple declaration of the files and directories a Task needs from their eventual "location" inside the Task Steps' containers.
• Provide a kind of "structural typing" for the contents of a workspace, and a way for different but compatible "types" to interface by mapping paths.

Goals

1. Provide a way for Task authors to declare paths expected on a workspace.
2. Provide a way for Task authors to declare paths that will be produced on a workspace.
3. Provide a way for Pipeline authors to declare important files that one PipelineTask produces and another expects.
4. Allow overriding the paths in TaskRuns and Pipelines/PipelineRuns.
5. Provide clear errors when expected or created paths are missing.
6. Provide a blessed way of declaring paths on a workspace in catalog tasks.
7. Inject new variables that allow Task and Pipeline authors to access paths without hard-coding them.

Non-Goals

• Declaring any other file features like file types, permissions, content validation, etc...

Requirements

• When a workspace is optional, files should only be validated if the workspace is provided.

• Any variables exposed by Workspace Paths should degrade gracefully if the workspace is optional and not provided. $(workspaces.foo.files.bar.path) should resolve to an empty string if the optional workspace foo is not provided.

• A Workspace Path can be any filesystem entry with a path: a file, a directory, a symlink, etc.

User Stories

Clearly declare credential requirements for a git Task

As a git Task author I want to declare which files my Task expects to appear in the ssh-credentials workspace so that users can quickly learn the necessary files to provide in their Secrets.

Validate that webpack.config.js Build configuration is passed

As a webpack build Task author I want to validate that webpack.config.js is present in a given workspace before I try to run webpack so that my Task fails early with a clear error message if an incorrectly populated workspace has been passed to it.

Enable arbitrary paths to contain package main

As the author of a go-build Task I want the user to be able to declare a specific directory to build inside a workspace so that projects with any directory structure can be built by my Task.

Team writing TaskRuns has specific requirements for directory structure of config-as-code

As the author of a TaskRun that builds and releases a project using "config-as-code" principles I want to organize my config and credential files in my team's repo according to my org's requirements (and not according to the requirements of the catalog Tasks I am choosing) so that I am structuring my projects within my company's agreed-upon guidelines.

References

Related Issue

• Some of the features stem from requirements in the issue Improve UX of getting credentials into Tasks.

Related Designs

• Echoes some of the described features of the FileSet Resource in the PipelineResources redesign from winter 2019/20.
• File contract features of the PipelineResources revamp.

Catalog Examples

Here are all of the examples in the catalog where we ask for both a workspace and a param providing a path into that workspace.

1. 02-build-runtime-with-gradle
2. 03-build-shared-class-cache
3. 04-finalize-runtime-with-function
4. 01-install-deps
5. 02-build-archive
6. 03-openwhisk
7. 01-install-deps
8. 02-build-archive
9. 03-openwhisk
10. ansible-runner
11. build-push-gke-deploy
12. buildpacks-phases
13. buildpacks
14. create-github-release
15. gcs-create-bucket
16. gcs-delete-bucket
17. gcs-download
18. gcs-generic
19. gcs-upload
20. git-batch-merge
21. git-batch-merge
22. git-clone
23. git-clone
24. github-app-token
25. gke-cluster-create
26. jib-gradle
27. jib-maven
28. kaniko
29. maven
30. wget