Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Explain how to use functions from libraries #2798

Merged
merged 1 commit into from
Jul 4, 2023
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 32 additions & 3 deletions docs/userguide/apps.rst
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,6 @@ The following alternative formulation is valid Parsl.

.. code-block:: python

import random
factor = 5

@python_app
Expand All @@ -71,7 +70,9 @@ In this case, Parsl will establish a dependency between the two apps and will no
execute the dependent app until all dependent futures are resolved.
Further detail is provided in :ref:`label-futures`.

A Python app may also act upon files. In order to make Parsl aware of these files, they must be specified by using the ``inputs`` and/or ``outputs`` keyword arguments, as in following code snippet, which copies the contents of one file (``in.txt``) to another (``out.txt``).
A Python app may also act upon files. In order to make Parsl aware of these files,
they must be specified by using the ``inputs`` and/or ``outputs`` keyword arguments,
as in following code snippet, which copies the contents of one file (``in.txt``) to another (``out.txt``).

.. code-block:: python

Expand All @@ -98,6 +99,34 @@ Any Parsl app (a Python function decorated with the ``@python_app`` or ``@bash_a
3. walltime: (int) This keyword argument places a limit on the app's
runtime in seconds. If the walltime is exceed, Parsl will raise an `parsl.app.errors.AppTimeout` exception.

Serializing Functions from Libraries
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Parsl can create Apps directly from functions defined in Python modules.
Supply the function as an argument to ``python_app`` rather than creating a new function which is decorated.

.. code-block:: python

from module import function
function_app = python_app(function, executors='all')

``function_app`` will act as Parsl App function of ``function``.

It is also possible to create wrapped versions of functions, such as ones with pinned arguments.
Parsl just requires first calling :meth:`~functools.update_wrapped` with the wrapped function
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what happens if you don't do this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I should test to be sure. I think we get errors when a function lacks a name, but might be confused with those errors originating from a code which uses Parsl and not Parsl itself.

to include attributes from the original function (e.g., its name).

.. code-block:: python

from functools import partial, update_wrapped
import numpy as np
my_max = partial(np.max, axis=0, keepdims=True)
my_max = update_wrapper(my_max, max) # Copy over the names
my_max_app = python_app(my_max)


Apps created using ``python_app`` as a function will work just like those which use it as a decorator.

Returns
^^^^^^^

Expand All @@ -112,7 +141,7 @@ Limitations
There are some limitations on the Python functions that can be converted to apps:

1. Functions should act only on defined input arguments. That is, they should not use script-level or global variables.
2. Functions must explicitly import any required modules.
2. Functions must explicitly import any required modules if they are defined in script which starts Parsl.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i'm fairly certain there's more subtlety here...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How so?

I'm fairly sure it works this way for functions defined in other files. The main Parsl process will serialize a reference to the location of the function, and the worker will import those libraries as it loads in the function.

That said, my knowledge of Parsl's function serialization is empirical. I don't understand the underlying codebase, just how it has worked for me in practice.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pickle does that, but dill does more complex stuff (including changing behaviour depending on where on the filesystem your Other Files are located...) that is hard to keep in my head

3. Parsl uses dill and pickle to serialize Python objects to/from apps. Therefore, Parsl require that all input and output objects can be serialized by dill or pickle. See :ref:`label_serialization_error`.
4. STDOUT and STDERR produced by Python apps remotely are not captured.

Expand Down