You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I've noticed some limitations for implementing Schedulers.jl. Some are inherent and some are implementation details. Solving the latter might require doing this in Base: JuliaLang/julia#43649
I/O
A conceptually tricky bit for implementing scheduler outside Base at the moment is libuv-interop. It is already possible to use I/O in Schedulers.jl but what is non-blocking I/O in the native scheduler is blocking I/O in Schedulers.jl. For non-blocking I/O, the tasks need to sleep in the scheduler in Schedulers.jl, not in the native Julia scheduler. This requires implementing I/O APIs in Schedulers.jl by re-enqueuing the task to the "external queue" in the libuv callback whenever I/O is ready.
Nested schedulers
Schedulers.jl supports multiple schedulers to exist in a single process but not nesting schedulers (i.e., using a set of Schedulers.Task as a worker pool). Currently, the worker tasks of Schedulers.jl have to be "real" Julia tasks and not Schedulers.jl's tasks. However, I don't think this is a strict limitation and it should be possible to support nesting. I'd guess the performance is OK if the queue is not empty but waking up worker tasks require a cascade of wake ups for the outer schedulers which has a negative impact to the performance.
Julia fallbacks to copying stack when allocating dedicated stack for a task failed. This requires marking task sticky because the stack needs to be restored to the same per-thread (per-ptls) stack. In principle, this can be supported in Schedulers.jl by either adding sticky queues to the scheduler or using the native sticky queues when the stack allocation failed. But I couldn't make these approaches work so far. The former approach can be found in: #23
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
I've noticed some limitations for implementing Schedulers.jl. Some are inherent and some are implementation details. Solving the latter might require doing this in
Base
: JuliaLang/julia#43649I/O
A conceptually tricky bit for implementing scheduler outside
Base
at the moment is libuv-interop. It is already possible to use I/O in Schedulers.jl but what is non-blocking I/O in the native scheduler is blocking I/O in Schedulers.jl. For non-blocking I/O, the tasks need to sleep in the scheduler in Schedulers.jl, not in the native Julia scheduler. This requires implementing I/O APIs in Schedulers.jl by re-enqueuing the task to the "external queue" in the libuv callback whenever I/O is ready.Nested schedulers
Schedulers.jl supports multiple schedulers to exist in a single process but not nesting schedulers (i.e., using a set of
Schedulers.Task
as a worker pool). Currently, the worker tasks of Schedulers.jl have to be "real" Julia tasks and not Schedulers.jl's tasks. However, I don't think this is a strict limitation and it should be possible to support nesting. I'd guess the performance is OK if the queue is not empty but waking up worker tasks require a cascade of wake ups for the outer schedulers which has a negative impact to the performance.No public API for migrating tasks with
yieldto
It's not clear how to call
yieldto(task, ...)
multiple times from different threads. To workaround this, I'm guessing the hiddentask->tid
field in the Cstruct
at precompile time and usingunsafe_store!
to reset it to-1
("unset") before each (re)schedule.Stack copying fallback
This is tracked in the issue #27
Julia fallbacks to copying stack when allocating dedicated stack for a task failed. This requires marking task sticky because the stack needs to be restored to the same per-thread (per-ptls) stack. In principle, this can be supported in Schedulers.jl by either adding sticky queues to the scheduler or using the native sticky queues when the stack allocation failed. But I couldn't make these approaches work so far. The former approach can be found in: #23
Beta Was this translation helpful? Give feedback.
All reactions