Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Explicit mention that Ax Service API is recommended throughout docs #2453

Closed
wants to merge 1 commit into from
Closed

Explicit mention that Ax Service API is recommended throughout docs #2453

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
89 changes: 61 additions & 28 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,19 +17,33 @@ Adaptive experimentation is the machine-learning guided process of iteratively
exploring a (possibly infinite) parameter space in order to identify optimal
configurations in a resource-efficient manner. Ax currently supports Bayesian
optimization and bandit optimization as exploration strategies. Bayesian
optimization in Ax is powered by [BoTorch](https://github.com/facebookexternal/botorch),
a modern library for Bayesian optimization research built on PyTorch.
optimization in Ax is powered by
[BoTorch](https://github.com/facebookexternal/botorch), a modern library for
Bayesian optimization research built on PyTorch.

For full documentation and tutorials, see the [Ax website](https://ax.dev)

## Why Ax?

* **Versatility**: Ax supports different kinds of experiments, from dynamic ML-assisted A/B testing, to hyperparameter optimization in machine learning.
* **Customization**: Ax makes it easy to add new modeling and decision algorithms, enabling research and development with minimal overhead.
* **Production-completeness**: Ax comes with storage integration and ability to fully save and reload experiments.
* **Support for multi-modal and constrained experimentation**: Ax allows for running and combining multiple experiments (e.g. simulation with a real-world "online" A/B test) and for constrained optimization (e.g. improving classification accuracy without significant increase in resource-utilization).
* **Efficiency in high-noise setting**: Ax offers state-of-the-art algorithms specifically geared to noisy experiments, such as simulations with reinforcement-learning agents.
* **Ease of use**: Ax includes 3 different APIs that strike different balances between lightweight structure and flexibility. Using the most concise Loop API, a whole optimization can be done in just one function call. The Service API integrates easily with external schedulers. The most elaborate Developer API affords full algorithm customization and experiment introspection.
- **Versatility**: Ax supports different kinds of experiments, from dynamic
ML-assisted A/B testing, to hyperparameter optimization in machine learning.
- **Customization**: Ax makes it easy to add new modeling and decision
algorithms, enabling research and development with minimal overhead.
- **Production-completeness**: Ax comes with storage integration and ability to
fully save and reload experiments.
- **Support for multi-modal and constrained experimentation**: Ax allows for
running and combining multiple experiments (e.g. simulation with a real-world
"online" A/B test) and for constrained optimization (e.g. improving
classification accuracy without significant increase in resource-utilization).
- **Efficiency in high-noise setting**: Ax offers state-of-the-art algorithms
specifically geared to noisy experiments, such as simulations with
reinforcement-learning agents.
- **Ease of use**: Ax includes 3 different APIs that strike different balances
between lightweight structure and flexibility. The Service API (recommended
for the vast majority of use-cases) provides an extensive, robust, and
easy-to-use interface to Ax; the Loop API enables particularly concise usage;
and the Developer API enables advanced experimental and methodological
control.

## Getting Started

Expand Down Expand Up @@ -63,47 +77,58 @@ artificial evaluation function):
## Installation

### Requirements

You need Python 3.10 or later to run Ax.

The required Python dependencies are:

* [botorch](https://www.botorch.org)
* jinja2
* pandas
* scipy
* sklearn
* plotly >=2.2.1
- [botorch](https://www.botorch.org)
- jinja2
- pandas
- scipy
- sklearn
- plotly >=2.2.1

### Stable Version

#### Installing via pip

We recommend installing Ax via pip (even if using Conda environment):

```
conda install pytorch torchvision -c pytorch # OSX only (details below)
pip install ax-platform
```

Installation will use Python wheels from PyPI, available for [OSX, Linux, and Windows](https://pypi.org/project/ax-platform/#files).
Installation will use Python wheels from PyPI, available for
[OSX, Linux, and Windows](https://pypi.org/project/ax-platform/#files).

*Note*: Make sure the `pip` being used to install `ax-platform` is actually the one from the newly created Conda environment.
If you're using a Unix-based OS, you can use `which pip` to check.
_Note_: Make sure the `pip` being used to install `ax-platform` is actually the
one from the newly created Conda environment. If you're using a Unix-based OS,
you can use `which pip` to check.

*Recommendation for MacOS users*: PyTorch is a required dependency of BoTorch, and can be automatically installed via pip.
However, **we recommend you [install PyTorch manually](https://pytorch.org/get-started/locally/#anaconda-1) before installing Ax, using the Anaconda package manager**.
Installing from Anaconda will link against MKL (a library that optimizes mathematical computation for Intel processors).
This will result in up to an order-of-magnitude speed-up for Bayesian optimization, as at the moment, installing PyTorch from pip does not link against MKL.
_Recommendation for MacOS users_: PyTorch is a required dependency of BoTorch,
and can be automatically installed via pip. However, **we recommend you
[install PyTorch manually](https://pytorch.org/get-started/locally/#anaconda-1)
before installing Ax, using the Anaconda package manager**. Installing from
Anaconda will link against MKL (a library that optimizes mathematical
computation for Intel processors). This will result in up to an
order-of-magnitude speed-up for Bayesian optimization, as at the moment,
installing PyTorch from pip does not link against MKL.

If you need CUDA on MacOS, you will need to build PyTorch from source. Please consult the PyTorch installation instructions above.
If you need CUDA on MacOS, you will need to build PyTorch from source. Please
consult the PyTorch installation instructions above.

#### Optional Dependencies

To use Ax with a notebook environment, you will need Jupyter. Install it first:

```
pip install jupyter
```

If you want to store the experiments in MySQL, you will need SQLAlchemy:

```
pip install SQLAlchemy
```
Expand All @@ -116,7 +141,8 @@ You can install the latest (bleeding edge) version from Git.

First, see recommendation for installing PyTorch for MacOS users above.

At times, the bleeding edge for Ax can depend on bleeding edge versions of BoTorch (or GPyTorch). We therefore recommend installing those from Git as well:
At times, the bleeding edge for Ax can depend on bleeding edge versions of
BoTorch (or GPyTorch). We therefore recommend installing those from Git as well:

```
pip install git+https://github.com/cornellius-gp/linear_operator.git
Expand All @@ -141,9 +167,11 @@ To support plotly-based plotting in newer Jupyter notebook versions
pip install "notebook>=5.3" "ipywidgets==7.5"
```

[See Plotly repo's README](https://github.com/plotly/plotly.py#jupyter-notebook-support) for details and JupyterLab instructions.
[See Plotly repo's README](https://github.com/plotly/plotly.py#jupyter-notebook-support)
for details and JupyterLab instructions.

If storing Ax experiments via SQLAlchemy in MySQL or SQLite:

```
pip install git+https://github.com/facebook/Ax.git#egg=ax-platform[mysql]
```
Expand All @@ -152,13 +180,18 @@ pip install git+https://github.com/facebook/Ax.git#egg=ax-platform[mysql]

### Getting help

Please open an issue on our [issues page](https://github.com/facebook/Ax/issues) with any questions, feature requests or bug reports! If posting a bug report, please include a minimal reproducible example (as a code snippet) that we can use to reproduce and debug the problem you encountered.
Please open an issue on our [issues page](https://github.com/facebook/Ax/issues)
with any questions, feature requests or bug reports! If posting a bug report,
please include a minimal reproducible example (as a code snippet) that we can
use to reproduce and debug the problem you encountered.

### Contributing

See the [CONTRIBUTING](CONTRIBUTING.md) file for how to help out.

When contributing to Ax, we recommend cloning the [repository](https://github.com/facebook/Ax) and installing all optional dependencies:
When contributing to Ax, we recommend cloning the
[repository](https://github.com/facebook/Ax) and installing all optional
dependencies:

```
pip install git+https://github.com/cornellius-gp/linear_operator.git
Expand All @@ -175,8 +208,8 @@ See recommendation for installing PyTorch for MacOS users above.

The above example limits the cloned directory size via the
[`--depth`](https://git-scm.com/docs/git-clone#Documentation/git-clone.txt---depthltdepthgt)
argument to `git clone`. If you require the entire commit history you may remove this
argument.
argument to `git clone`. If you require the entire commit history you may remove
this argument.

## License

Expand Down
7 changes: 4 additions & 3 deletions docs/api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,10 @@ From most lightweight to fullest functionality, our APIs are:
optimization is complete. **Use this API only for the simplest use cases where
running a single trial is fast and only one trial should be running at a
time.**
- **Service API** ([tutorial](/tutorials/gpei_hartmann_service.html)) can be
used as a lightweight service for parameter-tuning applications where trials
might be evaluated in parallel and data is available asynchronously (e.g.
- **[RECOMMENDED] Service API**
([tutorial](/tutorials/gpei_hartmann_service.html)) can be used as a
lightweight service for parameter-tuning applications where trials might be
evaluated in parallel and data is available asynchronously (e.g.
hyperparameter or simulation optimization). It requires little to no knowledge
of Ax data structures and easily integrates with various schedulers. In this
mode, Ax suggests one-[arm](glossary.md#arm) trials to be evaluated by the
Expand Down
100 changes: 79 additions & 21 deletions docs/trial-evaluation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,21 @@ id: trial-evaluation
title: Trial Evaluation
---

There are 3 paradigms for evaluating [trials](glossary.md#trial) in Ax.
Note: ensure that you are using the [appropriate type of trials](core.md#trial-vs-batched-trial) for your experiment, before proceeding to trial evaluation.

## Service API
The [```AxClient```](/api/service.html#module-ax.service.ax_client) exposes [```get_next_trial```](/api/service.html#ax.service.ax_client.AxClient.get_next_trial), as well as [```complete_trial```](/api/service.html#ax.service.ax_client.AxClient.complete_trial). The user is responsible for evaluating the trial parameters and passing the results to [```complete_trial```](/api/service.html#ax.service.ax_client.AxClient.complete_trial).
There are 3 paradigms for evaluating [trials](glossary.md#trial) in Ax. Note:
ensure that you are using the
[appropriate type of trials](core.md#trial-vs-batched-trial) for your
experiment, before proceeding to trial evaluation.

## [RECOMMENDED] Service API

The Service API [`AxClient`](/api/service.html#module-ax.service.ax_client)
exposes
[`get_next_trial`](/api/service.html#ax.service.ax_client.AxClient.get_next_trial),
as well as
[`complete_trial`](/api/service.html#ax.service.ax_client.AxClient.complete_trial).
The user is responsible for evaluating the trial parameters and passing the
results to
[`complete_trial`](/api/service.html#ax.service.ax_client.AxClient.complete_trial).

```python
...
Expand All @@ -19,16 +29,29 @@ for i in range(25):

### Evaluating Trial Parameters

In the Service API, the [```complete_trial```](/api/service.html#ax.service.ax_client.AxClient.complete_trial) requires `raw_data` evaluated from the parameters suggested by [```get_next_trial```](/api/service.html#ax.service.ax_client.AxClient.get_next_trial).
In the Service API, the
[`complete_trial`](/api/service.html#ax.service.ax_client.AxClient.complete_trial)
method requires `raw_data` evaluated from the parameters suggested by
[`get_next_trial`](/api/service.html#ax.service.ax_client.AxClient.get_next_trial).

The data can be in the form of:

- A dictionary of metric names to tuples of (mean and [SEM](glossary.md#sem))
- A single (mean, SEM) tuple
- A single mean

In the second case, Ax will assume that the mean and the SEM are for the experiment objective (if the evaluations are noiseless, simply provide an SEM of 0.0). In the third case, Ax will assume that observations are corrupted by Gaussian noise with zero mean and unknown SEM, and infer the SEM from the data (this is equivalent to specifying an SEM of None). Note that if the observation noise is non-zero (either provided or inferred), the "best arm" suggested by Ax may not always be the one whose evaluation returned the best observed value (as the "best arm" is selected based on the model-predicted mean).
In the second case, Ax will assume that the mean and the SEM are for the
experiment objective (if the evaluations are noiseless, simply provide an SEM of
0.0). In the third case, Ax will assume that observations are corrupted by
Gaussian noise with zero mean and unknown SEM, and infer the SEM from the data
(this is equivalent to specifying an SEM of None). Note that if the observation
noise is non-zero (either provided or inferred), the "best arm" suggested by Ax
may not always be the one whose evaluation returned the best observed value (as
the "best arm" is selected based on the model-predicted mean).

For example, this evaluation function computes mean and SEM for [Hartmann6](https://www.sfu.ca/~ssurjano/hart6.html) function and for the L2-norm. We return `0.0` for SEM since the observations are noiseless:
For example, this evaluation function computes mean and SEM for
[Hartmann6](https://www.sfu.ca/~ssurjano/hart6.html) function and for the
L2-norm. We return `0.0` for SEM since the observations are noiseless:

```python
from ax.utils.measurement.synthetic_functions import hartmann6
Expand All @@ -38,7 +61,9 @@ def hartmann_evaluation_function(parameterization):
return {"hartmann6": (hartmann6(x), 0.0), "l2norm": (np.sqrt((x ** 2).sum()), 0.0)}
```

This function computes just the objective mean and SEM, assuming the [Branin](https://www.sfu.ca/~ssurjano/branin.html) function is the objective of the experiment:
This function computes just the objective mean and SEM, assuming the
[Branin](https://www.sfu.ca/~ssurjano/branin.html) function is the objective of
the experiment:

```python
from ax.utils.measurement.synthetic_functions import branin
Expand All @@ -62,16 +87,32 @@ def branin_evaluation_function_unknown_sem(parameterization):
```

## Loop API
The [```optimize```](/api/service.html#ax.service.managed_loop.optimize) function requires an `evaluation_function`, which accepts parameters and returns raw data in the format described above.
It can also accept a `weight` parameter, a nullable `float` representing the fraction of available data on which the parameterization should be evaluated. For example, this could be a downsampling rate in case of hyperparameter optimization (what portion of data the ML model should be trained on for evaluation) or the percentage of users exposed to a given configuration in A/B testing. This weight is not used in unweighted experiments and defaults to `None`.

The [`optimize`](/api/service.html#ax.service.managed_loop.optimize) function
requires an `evaluation_function`, which accepts parameters and returns raw data
in the format described above. It can also accept a `weight` parameter, a
nullable `float` representing the fraction of available data on which the
parameterization should be evaluated. For example, this could be a downsampling
rate in case of hyperparameter optimization (what portion of data the ML model
should be trained on for evaluation) or the percentage of users exposed to a
given configuration in A/B testing. This weight is not used in unweighted
experiments and defaults to `None`.

## Developer API

The Developer API is supported by the [```Experiment```](/api/core.html#module-ax.core.experiment) class. In this paradigm, the user specifies:
* [`Runner`](../api/core.html#ax.core.runner.Runner): Defines how to deploy the experiment.
* List of [`Metrics`](../api/core.html#ax.core.metric.Metric): Each defines how to compute/fetch data for a given objective or outcome.
The Developer API is supported by the
[`Experiment`](/api/core.html#module-ax.core.experiment) class. In this
paradigm, the user specifies:

- [`Runner`](../api/core.html#ax.core.runner.Runner): Defines how to deploy the
experiment.
- List of [`Metrics`](../api/core.html#ax.core.metric.Metric): Each defines how
to compute/fetch data for a given objective or outcome.

The experiment requires a `generator_run` to create a new trial or batch trial.
A generator run can be generated by a model. The trial then has its own `run`
and `mark_complete` methods.

The experiment requires a `generator_run` to create a new trial or batch trial. A generator run can be generated by a model. The trial then has its own `run` and `mark_complete` methods.
```python
...
sobol = Models.SOBOL(exp.search_space)
Expand All @@ -90,7 +131,12 @@ for i in range(15):

### Custom Metrics

Similar to a trial evaluation in the Service API, a custom metric computes a mean and SEM for each arm of a trial. However, the metric's `fetch_trial_data` method will be called automatically by the experiment's [```fetch_data```](/api/core.html#ax.core.base_trial.BaseTrial.fetch_data) method. If there are multiple objectives or outcomes that need to be optimized for, each needs its own metric.
Similar to a trial evaluation in the Service API, a custom metric computes a
mean and SEM for each arm of a trial. However, the metric's `fetch_trial_data`
method will be called automatically by the experiment's
[`fetch_data`](/api/core.html#ax.core.base_trial.BaseTrial.fetch_data) method.
If there are multiple objectives or outcomes that need to be optimized for, each
needs its own metric.

```python
class MyMetric(Metric):
Expand All @@ -110,11 +156,23 @@ class MyMetric(Metric):

### Adding Your Own Runner

In order to control how the experiment is deployed, you can add your own runner. To do so, subclass [`Runner`](../api/core.html#ax.core.runner.Runner) and implement the [`run`](../api/core.html#ax.core.runner.Runner.run) method and [`staging_required`](../api/core.html#ax.core.runner.Runner.staging_required) property.

The [`run`](../api/core.html#ax.core.runner.Runner.run) method accepts a [`Trial`](../api/core.html#ax.core.trial.Trial) and returns a JSON-serializable dictionary of any necessary tracking info to fetch data later from this external system. A unique identifier or name for this trial in the external system should be stored in this dictionary with the key `"name"`, and this can later be accessed via `trial.deployed_name`.

The [`staging_required`](../api/core.html#ax.core.runner.Runner.staging_required) indicates whether the trial requires an intermediate staging period before evaluation begins. This property returns False by default.
In order to control how the experiment is deployed, you can add your own runner.
To do so, subclass [`Runner`](../api/core.html#ax.core.runner.Runner) and
implement the [`run`](../api/core.html#ax.core.runner.Runner.run) method and
[`staging_required`](../api/core.html#ax.core.runner.Runner.staging_required)
property.

The [`run`](../api/core.html#ax.core.runner.Runner.run) method accepts a
[`Trial`](../api/core.html#ax.core.trial.Trial) and returns a JSON-serializable
dictionary of any necessary tracking info to fetch data later from this external
system. A unique identifier or name for this trial in the external system should
be stored in this dictionary with the key `"name"`, and this can later be
accessed via `trial.deployed_name`.

The
[`staging_required`](../api/core.html#ax.core.runner.Runner.staging_required)
indicates whether the trial requires an intermediate staging period before
evaluation begins. This property returns False by default.

An example implementation is given below:

Expand Down
Loading