Explain how to use functions from libraries

[WardLT]

Description

The documentation does not explain that you can create Apps outside of the decorator-based mechanism.
It's a convenient route when you just want to make a "Parsl version" of a function already in a module,
and create Parsl functions programatically.

Type of change

• Documentation update

Context

Was reviewing code from a user and saw functions similar to:

@python_app
def f(x): 
   from m import f
   return f(x)

I suggested they replace with

from m import f
f = python_app(f)

to make the code more succinct.

It wasn't apparent in the docs that you could do this. I know you can because I'm familiar with function serialization.

[benclifford]

i'm fairly certain there's more subtlety here...

[WardLT]

How so?

I'm fairly sure it works this way for functions defined in other files. The main Parsl process will serialize a reference to the location of the function, and the worker will import those libraries as it loads in the function.

That said, my knowledge of Parsl's function serialization is empirical. I don't understand the underlying codebase, just how it has worked for me in practice.

[benclifford]

pickle does that, but dill does more complex stuff (including changing behaviour depending on where on the filesystem your Other Files are located...) that is hard to keep in my head

[benclifford]

what happens if you don't do this?

[WardLT]

I should test to be sure. I think we get errors when a function lacks a name, but might be confused with those errors originating from a code which uses Parsl and not Parsl itself.