Validate conditions for affinity assistant to operate correctly

[ghost]

Expected Behavior

Tekton validates whether a TaskRun's or PipelineRun's volume configuration will be impossible to manage with the Affinity Assistant.

Actual Behavior

Currently it's possible to get into a situation where tasks wait forever to start due to use of multiple persistent volumes as workspaces. Ideally we should validate that this will happen and either a) reject the taskrun / pipelinerun with a clear validation error or b) fail the runs early with a clear error reason + message.

Steps to Reproduce the Problem

1. See steps outlined in Task with more than one workspace waiting forever to start #2829

[ghost]

I wonder if there is another option, c) decide that using the affinity assistant for the multiple-workspace run will not work and fallback to running without it, emitting a warning.

[jlpettersson]

using the affinity assistant for the multiple-workspace run

multiple-workspace run will work with the affinity assistant - as long as multiple workspaces is not used by the same task.

and multi-PVC-tasks should not be used in Tasks - they will most likely deadlock in regional clusters sometimes as what happened in #2546

[ghost]

nice, thanks for clarifying!

[jlpettersson]

I can have a look at this.

/assign

[lmserrano]

I'm facing this issue for 2 nodes in a same single AZ and a pipelinerun which uses multiple pvcs and some tasks which also use multiple pvcs.

@jlpettersson , what is the suggested approach for avoiding multi-PVC tasks? In my use case, as part of a pipeline, I have a task which builds an application, and it does so based on 3 volumes: 1 with the git code and 2 with dependencies (which are retained across runs). Even if I put the dependencies together in a single pvc, I still can't make it work with less than 2, so I could use some guidance on how to best handle this. Thank you.

[jlpettersson]

@jlpettersson , what is the suggested approach for avoiding multi-PVC tasks?

As a generic solution, I suggests using subPaths of the same PVC volume. See the example with different subPaths: https://github.com/tektoncd/pipeline/blob/master/examples/v1beta1/pipelineruns/pipelinerun-using-different-subpaths-of-workspace.yaml

In my use case, as part of a pipeline, I have a task which builds an application, and it does so based on 3 volumes: 1 with the git code and 2 with dependencies (which are retained across runs). Even if I put the dependencies together in a single pvc, I still can't make it work with less than 2, so I could use some guidance on how to best handle this. Thank you.

There is probably multiple different solutions for this. It also depends on what storage solutions is available for you.

2 with dependencies (which are retained across runs)

Do you keep the volume, and mount it in access mode ReadOnlyMany in this case, or do you populate a new volume with the same dependencies for every run?

As a generic suggestion for this, I would recommend to store your dependencies in a bucket e.g. GCS or S3, when a new PipelineRun is initiated, I would use two parallel tasks:

• git-clone that clones the source files to a subPath of your workspace, e.g. subPath "src"
• extract-dependencies that downloads the dependencies from a bucket, GCS or S3 to another subPath of the same workspace volume, e.g. subPath "deps" or something that your build system prefer.

the following tasks, can see both source files and dependencies from different directories of the workspace, or they can use two workspaces with the same subPaths as the two initial tasks.

[lmserrano]

As a generic solution, I suggests using subPaths of the same PVC volume. See the example with different subPaths: https://github.com/tektoncd/pipeline/blob/master/examples/v1beta1/pipelineruns/pipelinerun-using-different-subpaths-of-workspace.yaml

Thank you. With that approach I can have the 2 dependency volumes be used in a single one.

There is probably multiple different solutions for this. It also depends on what storage solutions is available for you.

I'm currently using AWS default EBS which I know can pose issues when nodes go across availability zones, but for now I'm having my tekton pipelines and tasks in a single availability zone to avoid volume affinity handling complexity.

Do you keep the volume, and mount it in access mode ReadOnlyMany in this case, or do you populate a new volume with the same dependencies for every run?

Currently I'm using the PVCs with access mode of ReadWriteOnce. My use case is, I setup the project's CI/CD in tekton and then I do a first build of the application through a pipelinerun which will have to download maven and ivy2 dependencies, to the dependencies PVC which has a reclaimPolicy of Retain. The first build will take long(er), but if in subsequent builds (future invocations of the same pipelinerun) I can reuse the same PVC, then I will make the build process much quicker because the dependencies will be already there and if there are any new ones, I'll only have to download the (hopefully few) new ones.

As a generic suggestion for this, I would recommend to store your dependencies in a bucket e.g. GCS or S3, when a new PipelineRun is initiated, I would use two parallel tasks:

• git-clone that clones the source files to a subPath of your workspace, e.g. subPath "src"
• extract-dependencies that downloads the dependencies from a bucket, GCS or S3 to another subPath of the same workspace volume, e.g. subPath "deps" or something that your build system prefer.

the following tasks, can see both source files and dependencies from different directories of the workspace, or they can use two workspaces with the same subPaths as the two initial tasks.

Thank you for the suggestion. It's nice that there are ways to work around this, although I hope that Tekton continues to support multi-PVC tasks because they are powerful and simpler to use, and if there are situations which can cause deadlocks in regional clusters they can probably still be addressed by continuing to improve the affinity assistant, its documentation, and the capabilities to further label/annotate/mark in other ways the tasks, pipelines and volumes, so that the affinity assistant can have a more complete view for influencing the pods scheduling and volume binding, or even for the system to be able to give early feedback about potentially problematic configurations.