Is there a reason not to silently ignore secedes/rejoins when not on a worker?

[ms7463]

For example lets say we had the following code:

def pizza_dough(): return 'pizza_dough'  # pep8 violation to save space
def sauce(): return 'sauce'
def cheese(): return 'cheese'

def pizza():
    client = get_client()
    futs = [client.submit(ingredient) for ingredient in [pizza_dough, sauce, cheese]]
    secede()
    result = '+'.join(client.gather(futs))
    rejoin()
    return result

def bagel_bites():
    client = get_client()
    futs = [client.submit(ingredient) for ingredient in [bagel, pizza]]
    secede()
    bagel, pizza = '+'.join(client.gather(futs))
    rejoin()
    result = bagel + pizza.split('+', 1)[-1]
    return result

If I called pizza() as the root call. I would get an error due to the secede/rejoin not being called on a worker. To get around this I would need to put error handling around the secede/rejoin calls to make it work as a root or nested call, which could get verbose and unwieldy. As far as I can tell this is unnecessary and that logic could just be built into secede/rejoin directly. Is there already a way to do this? and if not is there any reason not to do so?

[fjetter]

That's a question about API design which is ultimately always also a matter of taste so this answer is subjective.

I believe an important factor in API design is the principle of least surprise which boils down to "An API should do what most users would anticipate it to do". An exception is usually raised if an assumption is invalidated or the expected result cannot be guaranteed for any other reason.

The secede and rejoin API calls are functions to remove (secede) a thread from the given worker thread pool such that it does not block an executing slot. Rejoin will revert this.
By this argumentation, calling secede in a thread which is not running on a worker both invalidates the assumption that there is a worker pool and will not be able to ensure the expected result (there is no spoon threadpool). From my POV that's a classical "raise Exception" scenario.

For you scenario, I would recommend to be more explicit about whether or not a function is supposed to be called on a worker or not. If you know, for a fact, that you are running on a worker and want to secede, I can also recommend looking into https://distributed.dask.org/en/latest/api.html?highlight=get_worker_client#distributed.worker_client

[ms7463]

That makes sense, thanks for the insight @fjetter.