You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
use gnumake jobclient to automatically grow and shrink the pool size
to limit the overall cpu load in a build system (make, cmake, ninja, ...)
this is useful on large build servers (example: 128 cores)
but the build system runs with make -j32 to limit cpu load
by default billiard.Pool(1000) and multiprocessing.Pool(1000)
will start 1000 workers *before* demand
when a jobserver is found, pool should start only one worker, and grow the pool *on* demand
for every extra worker request a token from the jobserver.
if the jobserver is full, dont start new workers
this can only grow the pool on demand
ideally it should also shrink the pool when workers run idle
→ release tokens to the jobserver, so other workers can run
challenge:
release tokens on exception or sys.exit
otherwise other workers can hang forever
probably with sys.excepthook
ideally "someone" would write a better jobserver protocol
where the jobserver is actively monitoring the worker processes (could also monitor cpu load)
and when a worker crashes without releasing tokens, the jobserver assumes the tokens are released
could be based on celery or similar
The text was updated successfully, but these errors were encountered:
use gnumake jobclient to automatically grow and shrink the pool size
to limit the overall cpu load in a build system (make, cmake, ninja, ...)
this is useful on large build servers (example: 128 cores)
but the build system runs with
make -j32
to limit cpu loadby default
billiard.Pool(1000)
andmultiprocessing.Pool(1000)
will start 1000 workers *before* demand
when a jobserver is found, pool should start only one worker, and grow the pool *on* demand
for every extra worker request a token from the jobserver.
if the jobserver is full, dont start new workers
example implementation
blink-bindgen-limit-jobs-with-jobclient.patch
this can only grow the pool on demand
ideally it should also shrink the pool when workers run idle
→ release tokens to the jobserver, so other workers can run
challenge:
release tokens on exception or sys.exit
otherwise other workers can hang forever
probably with sys.excepthook
ideally "someone" would write a better jobserver protocol
where the jobserver is actively monitoring the worker processes (could also monitor cpu load)
and when a worker crashes without releasing tokens, the jobserver assumes the tokens are released
could be based on celery or similar
The text was updated successfully, but these errors were encountered: