Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature / user-function input/output field decorators #1072

Merged
merged 18 commits into from
Dec 14, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
107 changes: 74 additions & 33 deletions docs/function_guides/generator.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,42 +5,84 @@ Generator Functions

Generator and :ref:`Simulator functions<funcguides-sim>` have relatively similar interfaces.

.. code-block:: python
Writing a Generator
-------------------

def my_generator(Input, persis_info, gen_specs, libE_info):
batch_size = gen_specs["user"]["batch_size"]
.. tab-set::

.. tab-item:: Non-decorated
:sync: nodecorate

.. code-block:: python

def my_generator(Input, persis_info, gen_specs, libE_info):
batch_size = gen_specs["user"]["batch_size"]

Output = np.zeros(batch_size, gen_specs["out"])
# ...
Output["x"], persis_info = generate_next_simulation_inputs(Input["f"], persis_info)

return Output, persis_info

.. tab-item:: Decorated
:sync: decorate

.. code-block:: python

Output = np.zeros(batch_size, gen_specs["out"])
...
Output["x"], persis_info = generate_next_simulation_inputs(Input["f"], persis_info)
from libensemble.specs import input_fields, output_data

return Output, persis_info

@input_fields(["f"])
@output_data([("x", float)])
def my_generator(Input, persis_info, gen_specs, libE_info):
batch_size = gen_specs["user"]["batch_size"]

Output = np.zeros(batch_size, gen_specs["out"])
# ...
Output["x"], persis_info = generate_next_simulation_inputs(Input["f"], persis_info)

return Output, persis_info

Most ``gen_f`` function definitions written by users resemble::

def my_generator(Input, persis_info, gen_specs, libE_info):

where:

* ``Input`` is a selection of the :ref:`History array<funcguides-history>`
* :ref:`persis_info<datastruct-persis-info>` is a dictionary containing state information
* :ref:`gen_specs<datastruct-gen-specs>` is a dictionary of generator parameters, including which fields from the History array got sent
* ``libE_info`` is a dictionary containing libEnsemble-specific entries
* ``Input`` is a selection of the :ref:`History array<funcguides-history>`, a NumPy array.
* :ref:`persis_info<datastruct-persis-info>` is a dictionary containing state information.
* :ref:`gen_specs<datastruct-gen-specs>` is a dictionary of generator parameters.
* ``libE_info`` is a dictionary containing miscellaneous entries.

Valid generator functions can accept a subset of the above parameters. So a very simple generator can start::

def my_generator(Input):

If gen_specs was initially defined::
If ``gen_specs`` was initially defined:

.. tab-set::

.. tab-item:: Non-decorated function
:sync: nodecorate

.. code-block:: python

gen_specs = GenSpecs(
gen_f=my_generator,
inputs=["f"],
outputs=["x", float, (1,)],
user={"batch_size": 128},
)

.. tab-item:: Decorated function
:sync: decorate

.. code-block:: python

gen_specs = {
"gen_f": some_function,
"in": ["f"],
"out:" ["x", float, (1,)],
"user": {
"batch_size": 128
}
}
gen_specs = GenSpecs(
gen_f=my_generator,
user={"batch_size": 128},
)

Then user parameters and a *local* array of outputs may be obtained/initialized like::

Expand All @@ -56,10 +98,9 @@ Then return the array and ``persis_info`` to libEnsemble::

return Output, persis_info

Between the ``Output`` definition and the ``return``, any level and complexity
of computation can be performed. Users are encouraged to use the :doc:`executor<../executor/overview>`
to submit applications to parallel resources if necessary, or plug in components from
other libraries to serve their needs.
Between the ``Output`` definition and the ``return``, any computation can be performed.
Users can try an :doc:`executor<../executor/overview>` to submit applications to parallel
resources, or plug in components from other libraries to serve their needs.

.. note::

Expand All @@ -74,17 +115,17 @@ Persistent Generators
While non-persistent generators return after completing their calculation, persistent
generators do the following in a loop:

1. Receive simulation results and metadata; exit if metadata instructs
2. Perform analysis
3. Send subsequent simulation parameters
1. Receive simulation results and metadata; exit if metadata instructs.
2. Perform analysis.
3. Send subsequent simulation parameters.

Persistent generators don't need to be re-initialized on each call, but are typically
more complicated. The :doc:`APOSMM<../examples/aposmm>`
optimization generator function included with libEnsemble is persistent so it can
maintain multiple local optimization subprocesses based on results from complete simulations.
more complicated. The persistent :doc:`APOSMM<../examples/aposmm>`
optimization generator function included with libEnsemble maintains
local optimization subprocesses based on results from complete simulations.

Use ``gen_specs["persis_in"]`` to specify fields to send back to the generator throughout the run.
``gen_specs["in"]`` only describes the input fields when the function is **first called**.
Use ``GenSpecs.persis_in`` to specify fields to send back to the generator throughout the run.
``GenSpecs.inputs`` only describes the input fields when the function is **first called**.

Functions for a persistent generator to communicate directly with the manager
are available in the :ref:`libensemble.tools.persistent_support<p_gen_routines>` class.
Expand Down Expand Up @@ -159,7 +200,7 @@ a worker can be initiated in *active receive* mode by the allocation
function (see :ref:`start_only_persistent<start_only_persistent_label>`).
The persistent worker can then send and receive from the manager at any time.

Ensure there are no communication deadlocks in this mode. In manager--worker message exchanges, only the worker-side
Ensure there are no communication deadlocks in this mode. In manager-worker message exchanges, only the worker-side
receive is blocking by default (a non-blocking option is available).

Cancelling Simulations
Expand Down
93 changes: 68 additions & 25 deletions docs/function_guides/simulator.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,42 +5,86 @@ Simulator Functions

Simulator and :ref:`Generator functions<funcguides-gen>` have relatively similar interfaces.

.. code-block:: python
Writing a Simulator
-------------------

.. tab-set::

.. tab-item:: Non-decorated
:sync: nodecorate

.. code-block:: python

def my_simulation(Input, persis_info, sim_specs, libE_info):
batch_size = sim_specs["user"]["batch_size"]

Output = np.zeros(batch_size, sim_specs["out"])
# ...
Output["f"], persis_info = do_a_simulation(Input["x"], persis_info)

return Output, persis_info

.. tab-item:: Decorated
:sync: decorate

.. code-block:: python

from libensemble.specs import input_fields, output_data

def my_simulation(Input, persis_info, sim_specs, libE_info):
batch_size = sim_specs["user"]["batch_size"]

Output = np.zeros(batch_size, sim_specs["out"])
...
Output["f"], persis_info = do_a_simulation(Input["x"], persis_info)
@input_fields(["x"])
@output_data([("f", float)])
def my_simulation(Input, persis_info, sim_specs, libE_info):
batch_size = sim_specs["user"]["batch_size"]

Output = np.zeros(batch_size, sim_specs["out"])
# ...
Output["f"], persis_info = do_a_simulation(Input["x"], persis_info)

return Output, persis_info

return Output, persis_info

Most ``sim_f`` function definitions written by users resemble::

def my_simulation(Input, persis_info, sim_specs, libE_info):

where:

* ``Input`` is a selection of the :ref:`History array<funcguides-history>`
* :ref:`persis_info<datastruct-persis-info>` is a dictionary containing state information
* :ref:`sim_specs<datastruct-sim-specs>` is a dictionary of simulation parameters, including which fields from the History array got sent
* ``libE_info`` is a dictionary containing libEnsemble-specific entries
* ``Input`` is a selection of the :ref:`History array<funcguides-history>`, a NumPy array.
* :ref:`persis_info<datastruct-persis-info>` is a dictionary containing state information.
* :ref:`sim_specs<datastruct-sim-specs>` is a dictionary of simulation parameters.
* ``libE_info`` is a dictionary containing libEnsemble-specific entries.

Valid simulator functions can accept a subset of the above parameters. So a very simple simulator function can start::

def my_simulation(Input):

If sim_specs was initially defined::
If ``sim_specs`` was initially defined:

.. tab-set::

.. tab-item:: Non-decorated function
:sync: nodecorate

.. code-block:: python

sim_specs = SimSpecs(
sim_f=my_simulation,
inputs=["x"],
outputs=["f", float, (1,)],
user={"batch_size": 128},
)

.. tab-item:: Decorated function
:sync: decorate

.. code-block:: python

sim_specs = SimSpecs(
sim_f=my_simulation,
user={"batch_size": 128},
)

sim_specs = {
"sim_f": some_function,
"in": ["x"],
"out:" ["f", float, (1,)],
"user": {
"batch_size": 128
}
}

Then user parameters and a *local* array of outputs may be obtained/initialized like::

Expand All @@ -55,10 +99,9 @@ Then return the array and ``persis_info`` to libEnsemble::

return Output, persis_info

Between the ``Output`` definition and the ``return``, any level and complexity
of computation can be performed. Users are encouraged to use the :doc:`executor<../executor/overview>`
to submit applications to parallel resources if necessary, or plug in components from
other libraries to serve their needs.
Between the ``Output`` definition and the ``return``, any computation can be performed.
Users can try an :doc:`executor<../executor/overview>` to submit applications to parallel
resources, or plug in components from other libraries to serve their needs.

Executor
--------
Expand All @@ -73,7 +116,7 @@ for an additional example to try out.
Persistent Simulators
---------------------

Although comparatively uncommon, simulator functions can also be written
Simulator functions can also be written
in a persistent fashion. See the :ref:`here<persistent-gens>` for a general API overview
of writing persistent generators, since the interface is largely identical. The only
differences are to pass ``EVAL_SIM_TAG`` when instantiating a ``PersistentSupport``
Expand Down
4 changes: 4 additions & 0 deletions libensemble/sim_funcs/one_d_func.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,11 @@

import numpy as np

from libensemble.specs import input_fields, output_data


@input_fields(["x"])
@output_data([("f", float)])
def one_d_example(x, persis_info, sim_specs, _):
"""
Evaluates the six hump camel function for a single point ``x``.
Expand Down
3 changes: 3 additions & 0 deletions libensemble/sim_funcs/six_hump_camel.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,12 @@
import numpy as np

from libensemble.message_numbers import EVAL_SIM_TAG, FINISHED_PERSISTENT_SIM_TAG, PERSIS_STOP, STOP_TAG
from libensemble.specs import input_fields, output_data
from libensemble.tools.persistent_support import PersistentSupport


@input_fields(["x"])
@output_data([("f", float)])
def six_hump_camel(H, persis_info, sim_specs, libE_info):
"""
Evaluates the six hump camel function for a collection of points given in ``H["x"]``.
Expand Down
Loading
Loading