How to run TaskRuns in non-parallel mode

[xclud]

I am using Tekton for my CI/CD jobs. The TaskRuns are made by a gitlab trigger (https://github.com/tektoncd/triggers).

Since compiling the code and building the docker images use a lot of system resources (Specially RAM), I need to run the tasks in a sequential order, not in parallel. Also, it doesn't make sense if a later trigger deploys before a first trigger (which happens quite a lot).

I appreciate if you shine some light on this.

[nikhil-thomas]

I need to run the tasks in a sequential order, not in parallel.

this should be what you are looking for. 🧑‍💻
https://tekton.dev/docs/pipelines/pipelines/#using-the-runafter-parameter

[nikhil-thomas]

Also, it doesn't make sense if a later trigger deploys before a first trigger (which happens quite a lot).

@afrittoli @savitaashture could we suggest any best practices/tips for this.

[xclud]

@nikhil-thomas It seems runAfter works in a Pipeline. My aim is to execute TaskRuns one at a time. Or if I change my question: How to run one Pipeline at a time?

[nikhil-thomas]

looks like a you will have to write a Task, whose taskrun (generated by trigger event) will check whether a pipelinerun exists and then create the pipelinerun, may be using tkn pipeline start command.

cc @chmouel

[xclud]

I have a git repository with many users creating feature branches and pushing into master. Each time a commit is pushed it triggers a new TaskRun and too many of jobs eat the system resources and the system hangs. They also go into a race condition to deploy to the server.

In contrast, Gitlab's built-in CI/CD pipeline runs the jobs one by one (not too many of them at once) and the order of execution is the datetime of the created job. I need to achieve this with Tekton.

[chmouel]

It's kind of hard, i think there is a other issu1e somewhere where @imjasonh explains why it is so difficult to do this properly,

maybe a CustomTask backed by a Queue service may be a (non trivial and heavy) way to try to achieve this but we don't have a solution built in pipeline for this.

[ghost]

/kind question

[afrittoli]

One way you could address this is to create your PipelineRun in pending mode, and have an external controller running them at the rate you desire - the pending mode was introduced exactly for this purpose, to throttle parallel execution.

https://tekton.dev/docs/pipelines/pipelineruns/#pending-pipelineruns

The only gotcha here is that you'd have to use PipelineRun instead of TaskRun.
It should be possible to add pending to TaskRuns too as a new feature.

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale with a justification.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[jerop]

/remove-lifecycle stale

so we can explore supporting pending TaskRuns to address this issue

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale with a justification.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[tekton-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

[tekton-robot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen with a justification.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/close

Send feedback to tektoncd/plumbing.

[tekton-robot]

@tekton-robot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen with a justification.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/close

Send feedback to tektoncd/plumbing.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.