-
Notifications
You must be signed in to change notification settings - Fork 94
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
xtriggers: re-implement as async functions #3497
Comments
Supersedes #2917 ! |
Didn't spot that! |
I think this is blocking (hence why
However, there's an option to make it I may be wrong (?).. |
There'll be a way of making it non-blocking, it's an untested illustrative example. [edit] I think this example is non-blocking providing your socket implementation doesn't have additional blocking parts. But as an aside putting the xtrigger runner into another process wouldn't be a bad shout to reduce the potential for xtriggers blocking the Scheduler's main loop similar to how they can currently block the subprocpool's main loop by saturating the pool. |
Comment from a NIWA colleague:
|
Discussed another potential use case here recently where an integer cycling workflow is driven by an external system. |
Note, currently the only way to achieve external event driven cycling in Cylc is using (deprecated?) extriggers (not to be mistaken for xtriggers). # An example of a CI-type workflow which kicks off a new cycle of tasks every time
# an external-trigger is received
# Start the workflow as normal:
# $ cylc vip <path/to/this/workflow>
# Then kick off a cycle specifying any desired environment variables e.g:
# $ ./bin/trigger <workflow-id> WORLD=earth
[scheduling]
cycling mode = integer
initial cycle point = 1
runahead limit = P5 # max number of cycles which can run in parallel
[[special tasks]]
# register the external trigger and tell it which task to run
external-trigger = build("build")
[[graph]]
P1 = """
build => a => b => c
"""
[runtime]
[[build, a, b, c]]
script = """
echo "Hello $CYLC_TASK_NAME!"
""" |
An idea I've been mulling for a while, for discussion when this issue is picked up... Implement these async xtriggers as tasks, not xtriggers, i.e: [runtime]
[[@file_watcher]] # @ prefix not necessary
run mode = xtrigger # using Tim's lovely new "run mode" feature developed for skip mode
[[[xtrigger]]]
function = cylc.site_extensions.ops_file_watcher
# xtrigger args
cycle = ops/global//<cycle>
mode = operational
[[[environment]]]
ANSWER = 42
[[[outputs]]]
file1 = file1
file2 = file2
file3 = file3 Why:
What this doesn't solve (passively) is event-driven cycling (see 1, 2). This is a problem async xtriggers would be perfectly capable of solving which is currently only possible via [scheduling]
[[graph]]
@file_watcher = """
foo => bar => baz
""" But I haven't got any good suggestions for that ATM. |
Supersedes the same idea from #2917
The Problem:
At the moment XTriggers are run by the subprocess pool, consequences:
BotchingBodging the XTrigger loading mechanism for #3465 seems nasty.Close #2917
The Proposal:
Convert XTriggers to asynchronous functions and call them directly from the main loop piggy-backing on the main loop plugin functionality introduced in #3492.
This will turn XTriggers into a blocking stage of the main loop (where they would effectively be running in an unlimited thread pool) something like this:
Consequences:
SubProcPool
, Subprocess, Python andimportlib
.Bonus Marks:
If we move to asynchronous XTriggers we gain the ability to have long-running XTrigger functions with minimal overheads.
This means we can very easily and efficiently implement a push interface for XTriggers (in addition to the pre existing pill interface):
Super Bonus Marks:
Another long-lived interface which would also be fairly straight forward to implement is ‘yield’ I.e a single long-lived asynchronous function which yields cycle points and data as and when it becomes available:
A nicer solution for Kafka / message brokers?
Hyper bonus marks:
Once you’ve made it as far as a yield interface you have effectively achieved the cycle driver interface I’ve been harking on about for the last two years. Use coroutines to provide a cycling interface rather than
botchingbodging external triggers into a cycling regime which doesn’t fit, they can become a first class cycling object in their own right:This is a little more involved, requiring a major abstraction of the cycling interface and classes and beyond the scope of Cylc8 but worth keeping in mind so as to keep doors open.
Questions:
The backbone of the work has already been done with main loop plugins (XTriggers are just a special case) but are there any hitches @cylc/core ?
Pull requests welcome!
The text was updated successfully, but these errors were encountered: