Document how PVC access modes affect pipeline execution #2521

jlpettersson · 2020-05-01T10:22:44Z

Changes

The access mode configured for a PVC used as a workspace volume source
may affect how a pipeline is executed, this is not well documented. This commit
intend to improve the documentation about this and also provide an example
on how to run parallel tasks when using PVC with ReadWriteOnce access mode.

Document how access mode affect task ordering in pipeline under "Specifying Workspace order in a Pipeline"
Provide a full pipeline example of how to use parallel tasks when using a PVC with access mode ReadWriteOnce
We do not provide a "PipelineRun example" under "Example PipelineRun definitions using Workspaces". This commit provide a full PipelineRun example using a workspace. Instead of providing examples of different volume sources here, examples on how to use different VolumeSources is moved to "Specifying VolumeSources in Workspaces"
The VolumeSources persistentVolumeClaim and volumeClaimTemplate both is a PVC, and PVCs has its own peculiarities, e.g. access mode, that we now document. Both PVC volume sources is moved to a section so we can document the common peculiarities in a single place
Add the workspace variable introduced in Add variable substitution for PVC name #2506

Submitter Checklist

These are the criteria that every PR should meet, please check them off as you
review them:

Includes tests (if functionality changed/added)
Includes docs (if user facing)
Commit messages follow commit message best practices

See the contribution guide for more details.

Double check this list of stuff that's easy to miss:

If you are adding a new binary/image to the cmd dir, please update
the release Task to build and release this image.

Reviewer Notes

If API changes are included, additive changes must be approved by at least two OWNERS and backwards incompatible changes must be approved by more than 50% of the OWNERS, and they must first be added in a backwards compatible way.

/kind documentation

tekton-robot · 2020-05-01T10:22:59Z

Hi @jlpettersson. Thanks for your PR.

I'm waiting for a tektoncd member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

ghost

Thankyou for this additional documentation, really useful to know!

docs/workspaces.md

ghost · 2020-05-01T10:42:17Z

docs/workspaces.md

 information, see the [`runAfter` documentation](pipelines.md#runAfter).

+**Warning:** You *must* ensure that this order is compatible with your configured access modes for your `PersistentVolumeClaim`.
+Parallel `Tasks` using the same `PersistentVolumeClaim` with access mode `ReadWriteOnce`, may execute on
+different nodes and be forced to execute sequentially which may cause `Tasks` to time out.


Does the k8s scheduler force this sequential ordering automatically or is it something the user is forced to do by adding runAfter?

The k8s scheduler puts pods on nodes - immediately, they all mount the PVC. But the mount only succeeds on one node at the time. Pods on other nodes is stuck in state "Pending" - but they eventually start to run when the volume is not used on other nodes anymore.

And when the pods in the "other nodes" gets into "Running" - the Task is already started - so it may have "timed out" as was already documented.

But on my single node cluster - all tasks can run concurrently, so I did not see that parallel tasks on other nodes may not start until the volume is available.

I did not know how this was working, so I had to create a bigger test to validate this. Me and @skaegi discussed about it in the last API WG.

docs/workspaces.md

ghost · 2020-05-01T10:53:55Z

examples/v1beta1/pipelineruns/pipelinerun-using-different-subpaths-of-workspace.yaml

@@ -46,6 +46,8 @@ spec:
 workspace: ws
 subPath: dir-1
 - name: writer-2
+ runAfter:
+ - writer-1


Interesting! This example was working already I think? So here we are just kinda documenting the ordering which the PVC accessMode already forced?

Yes, it was "working" but the tasks was declared "parallel". So:

the two tasks will successfully run concurrently if scheduled to the same node

the two tasks may successfully run sequentially if scheduled to different nodes and the second task does not time out

the second task may time out if it is scheduled to another node and it takes too long time before it can execute/finish (I dont know how we implement time out)

Consequentially: If this is used in regression testing - the test may be flaky sometimes.

But this fixes so the two tasks always run in a sequence. For parallel tasks we should do as in the example provided in this commit. Unless we know that it will not time out.

Ah, I see, great catch!

ghost · 2020-05-01T11:18:26Z

/ok-to-test

jlpettersson · 2020-05-01T13:21:26Z

/test pull-tekton-pipeline-integration-tests

tekton-robot · 2020-05-01T13:21:33Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sbwsg

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [sbwsg]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

jlpettersson · 2020-05-01T16:21:23Z

I don't see why the integration tests fails, but I'll give it a new try...

/test pull-tekton-pipeline-integration-tests

jlpettersson · 2020-05-01T17:45:09Z

/test pull-tekton-pipeline-integration-tests

jlpettersson · 2020-05-01T18:24:17Z

/test pull-tekton-pipeline-integration-tests

The access mode configured for a PVC that is used as a workspace volume source may affect how a pipeline is executed, this is not well documented. This commit intend to improve the documentation about this and also provide an example on how to run parallel tasks when using PVC with ReadWriteOnce access mode. - Document how access mode affect task ordering in pipeline under "Specifying Workspace order in a Pipeline" - Provide a full pipeline example of how to use parallel tasks when using a PVC with access mode ReadWriteOnce - We we did provide a "PipelineRun example" under "Example PipelineRun definitions using Workspaces". This commit provide a full PipelineRun example using a workspace. Instead of providing examples of different volume sources here, examples on how to use different VolumeSources is moved to "Specifying VolumeSources in Workspaces" - The VolumeSources persistentVolumeClaim and volumeClaimTemplate both is a PVC, and PVCs has its own peculiarities, e.g. access mode that we document. Both PVC volume sources is moved to a section so we can document the common peculiarities in a single place - Add the workspace variable introduced in tektoncd#2506

jlpettersson · 2020-05-01T18:58:59Z

Error from server (AlreadyExists): error when creating "STDIN": tasks.tekton.dev "writer" already exists

There was actually a problem with naming in my example - name collision with another example file. I changed name on my example task.

jlpettersson · 2020-05-01T19:53:18Z

/test pull-tekton-pipeline-integration-tests

jlpettersson · 2020-05-01T21:24:31Z

/test pull-tekton-pipeline-integration-tests

jlpettersson · 2020-05-02T07:06:36Z

/test pull-tekton-pipeline-integration-tests

afrittoli

Thank you!
/lgtm

jlpettersson · 2020-05-02T08:57:11Z

/test pull-tekton-pipeline-build-tests

jlpettersson · 2020-05-02T09:06:17Z

/test pull-tekton-pipeline-integration-tests

jlpettersson · 2020-05-02T09:46:45Z

/test pull-tekton-pipeline-integration-tests

This commit contains similar changes for the TaskRun section as what was done for the PipelineRun section in tektoncd#2521 - We did not provide a full example of a TaskRun definition under "Examples of `TaskRun` definition using `Workspaces`" This commit add a full example of a TaskRun definition. - Examples of how to use different volume sources already exists under "Specifying `VolumeSources` in `Workspaces`", add a link to that section instead of having the same examples under the TaskRun section. - A few broken links in the table-of-contents are fixed. /kind documentation

This commit contains similar changes for the TaskRun section as what was done for the PipelineRun section in #2521 - We did not provide a full example of a TaskRun definition under "Examples of `TaskRun` definition using `Workspaces`" This commit add a full example of a TaskRun definition. - Examples of how to use different volume sources already exists under "Specifying `VolumeSources` in `Workspaces`", add a link to that section instead of having the same examples under the TaskRun section. - A few broken links in the table-of-contents are fixed. /kind documentation

This commit contains similar changes for the TaskRun section as what was done for the PipelineRun section in tektoncd#2521 - We did not provide a full example of a TaskRun definition under "Examples of `TaskRun` definition using `Workspaces`" This commit add a full example of a TaskRun definition. - Examples of how to use different volume sources already exists under "Specifying `VolumeSources` in `Workspaces`", add a link to that section instead of having the same examples under the TaskRun section. - A few broken links in the table-of-contents are fixed. /kind documentation

Jonas has recently become a regularly contributor. He started with adding a minor [_missing_ `omitempty`](tektoncd/pipeline#2301) and then [proposed some ideas](tektoncd/pipeline#1986 (comment)) around workspaces and PersistentVolumeClaim creation and continued to [elaborate around those ideas](tektoncd/pipeline#1986 (comment)). A sunny day a few days later, he also submitted an [extensive implementation for volumeClaimTemplate](tektoncd/pipeline#2326), corresponding to the idea discussions. A few days later submitted a [small refactoring PR](tektoncd/pipeline#2392), and he also listened to community members that [proposed changes](tektoncd/pipeline#2450) to his implementation about volumeClaimTemplates and did an [implementation for that proposal](tektoncd/pipeline#2453). A rainy day, he also wrote [technical documentation about PVCs](tektoncd/pipeline#2521) including adding an example that caused _flaky_ integration tests for the whole community during multiple days. When he understood his mistake, he submitted a [removal of the example](tektoncd/pipeline#2546) that caused flaky tests. He has also put his toe into Tekton Catalog and [contributed to the buildah task](tektoncd/pipeline#2546). This has followed, mostly with more PRs to the Pipeline project: - tektoncd/pipeline#2460 - tektoncd/pipeline#2491 - tektoncd/pipeline#2502 - tektoncd/pipeline#2506 - tektoncd/pipeline#2632 - tektoncd/pipeline#2633 - tektoncd/pipeline#2634 - tektoncd/pipeline#2636 - tektoncd/pipeline#2601 - tektoncd/pipeline#2630 Jonas is excited about the great community around Tekton and the project! He now would like to join the org.

tekton-robot added the kind/documentation Categorizes issue or PR as related to documentation. label May 1, 2020

tekton-robot requested review from dlorenc and a user May 1, 2020 10:22

tekton-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels May 1, 2020

ghost reviewed May 1, 2020

View reviewed changes

tekton-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels May 1, 2020

jlpettersson force-pushed the document_peculiarities_with_pvc branch from 2c17d70 to 38603ab Compare May 1, 2020 11:33

ghost approved these changes May 1, 2020

View reviewed changes

tekton-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 1, 2020

jlpettersson force-pushed the document_peculiarities_with_pvc branch from 38603ab to 4e8a7a5 Compare May 1, 2020 18:57

afrittoli reviewed May 2, 2020

View reviewed changes

tekton-robot assigned afrittoli May 2, 2020

tekton-robot added the lgtm Indicates that a PR is ready to be merged. label May 2, 2020

tekton-robot merged commit 7bbc7eb into tektoncd:master May 2, 2020

This was referenced May 4, 2020

yaml tests seem to be consistently timing out #2540

Closed

Remove example with parallel tasks with a fan-in task #2546

Merged

jlpettersson mentioned this pull request May 16, 2020

Improve TaskRun workspaces documentation #2632

Merged

3 tasks

jlpettersson mentioned this pull request May 19, 2020

Add Jonas to the org tektoncd/community#106

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Document how PVC access modes affect pipeline execution #2521

Document how PVC access modes affect pipeline execution #2521

jlpettersson commented May 1, 2020

tekton-robot commented May 1, 2020

ghost left a comment

ghost May 1, 2020

jlpettersson May 1, 2020

ghost May 1, 2020

jlpettersson May 1, 2020

ghost May 1, 2020

ghost commented May 1, 2020

jlpettersson commented May 1, 2020

tekton-robot commented May 1, 2020

jlpettersson commented May 1, 2020

jlpettersson commented May 1, 2020

jlpettersson commented May 1, 2020

jlpettersson commented May 1, 2020

jlpettersson commented May 1, 2020

jlpettersson commented May 1, 2020

jlpettersson commented May 2, 2020

afrittoli left a comment

jlpettersson commented May 2, 2020

jlpettersson commented May 2, 2020

jlpettersson commented May 2, 2020

Document how PVC access modes affect pipeline execution #2521

Document how PVC access modes affect pipeline execution #2521

Conversation

jlpettersson commented May 1, 2020

Changes

Submitter Checklist

Reviewer Notes

tekton-robot commented May 1, 2020

ghost left a comment

Choose a reason for hiding this comment

ghost May 1, 2020

Choose a reason for hiding this comment

jlpettersson May 1, 2020

Choose a reason for hiding this comment

ghost May 1, 2020

Choose a reason for hiding this comment

jlpettersson May 1, 2020

Choose a reason for hiding this comment

ghost May 1, 2020

Choose a reason for hiding this comment

ghost commented May 1, 2020

jlpettersson commented May 1, 2020

tekton-robot commented May 1, 2020

jlpettersson commented May 1, 2020

jlpettersson commented May 1, 2020

jlpettersson commented May 1, 2020

jlpettersson commented May 1, 2020

jlpettersson commented May 1, 2020

jlpettersson commented May 1, 2020

jlpettersson commented May 2, 2020

afrittoli left a comment

Choose a reason for hiding this comment

jlpettersson commented May 2, 2020

jlpettersson commented May 2, 2020

jlpettersson commented May 2, 2020