Allow graceful shutdown, letting current builds finish

[vasi]

Prerequisites

I don't see any similar issue.

What is your proposal?

Provide a way to tell poudriere bulk to shut down, but only after waiting for existing builds to finish.

What is the existing behavior, if any?

Currently, Poudriere can be told to stop with Ctrl-C (SIGINT). This immediately stops any ongoing builds. Long builds like www/chromium will have to restart from scratch later.

What is the motivation / use case for the change?

I'm often in the middle of a large build, and realize I need to stop sometime soon. Maybe I need to build and install a small package, or need to restart the host for security updates, or want to change some poudriere settings before continuing.

This is fine when the current bulk-build involves many small packages, poudriere will only lose a small amount of build time if it's interrupted. But when I'm 20 hours into a www/chromium build, it's unpleasant to have to interrupt it and start that build from scratch.

Synth already has this feature, and others have requested it in the FreeBSD forums.

Did you consider any alternatives?

• One could wait until the large package is complete, and only then interrupt Poudriere. I guess a script could be written to do this? But it would be pretty unpleasant, especially since Poudriere would continue to start more parallel builds.
• For some packages, ccache mitigates the impact of stopping large builds before they're complete.

Is this really a ports feature request?

No, there is no way ports alone could implement this.

Example

N/A. Interface could be a new poudriere graceful command, or something like sending SIGUSR1 to poudriere to tell it to gracefully shutdown.

Additional context

None.

[bdrewery]

I like this and have been thinking of it myself. I think the first CTRL-c should prevent new builds. A second should terminate. I think the new jobs code can handle this. Probably 2 SIGTERM to do the same non-interactively.

[mirror176]

Having an adjustable state that could be increased/decreased would be valuable. Scheduler could be set to do all requested work, not schedule new dependency trees but finish the current ones, not schedule any new work but finish the current ones, and shutdown current work. All but that last state would be a state where it could keep being changed.; I presume people would be okay just restarting poudriere when the last is reached

Package repository could be updated in the second state if we moved package deletion to only when it or its dependency tree actually starts. Doing so for the third and fourth state would require tracking which packages were part of completed trees and then those trees alone could be updated into the package repository but suspect we need to keep the older deleted packages to mix together such a repository.

If successful partial repository tree updates happen then we could use that same logic to make pkg updates available before entire build runs finish without putting packages in an inconsistent state. Could be handy to be able to get out a security or other important/desired update without waiting for a full build to finish hours/days/etc. later. If supporting partial repository updates, we should review how to make the repository add/remove packages faster otherwise we need options to manually trigger it during a run or a limiter set so an update doesn't run for every branch completion.

In a basic state I presume this would not create an updated package repository only if there was still work to do but did not get scheduled; would be bad if sending the signal when the last packages were being built, or the last ones aborted due to failures of what is currently being built, could cause that otherwise 'finished' state to not create the package repository.

[zackattackz]

Anyone working on a PR for this? I'd be interested to make a draft if no one else is.

[zackattackz]

Probably 2 SIGTERM to do the same non-interactively.

One thing I'd be concerned with is potential breakage of scripts that send sigterm and expect it to stop immediately. They would break in the sense that they would silently continue to finish current jobs. Which could cause a headache for someone trying to figure out why poudriere is still running despite sending sigterm.