integrations: Optuna

[daavoo]

From Optuna:

Optuna is an automatic hyperparameter optimization software framework, particularly designed for machine learning. It features an imperative, define-by-run style user API. Thanks to our define-by-run API, the code written with Optuna enjoys high modularity, and the user of Optuna can dynamically construct the search spaces for the hyperparameters.

Existing integrations with other ML Loggers are maintained in the repository implemented as callbacks:

https://github.com/optuna/optuna/tree/master/optuna/integration

Related discord thread:

https://discordapp.com/channels/485586884165107732/563406153334128681/866102743235297280

[dberenbaum]

Interesting idea!

I imagine it would look something like study.optimize(objective, n_trials=10, callbacks=[DVCLiveCallback()]), and the output of each completed trial would be logged as a step with its corresponding metrics.

Some questions I have:

1. For DVC integration, would it be better to represent each trial as a checkpoint within a single experiment, or as individual experiments (in other words, should a trial include the full history of previous trials)? See nest_trials in https://github.com/optuna/optuna/blob/master/optuna/integration/mlflow.py. The previous trials influence the hyperparameters for the next trial, but each trial is an independent model.
2. How would trial hyperparameters be logged? Does this depend on logger: extend with log_param #100 to log parameters from each experiment? Should they be handled like metrics (outputs that get stored in .tsv)?
3. How to retrieve a specific trial? At least model weights/objects will need to be saved for each trial if I want to reuse the result of any trial. This doesn't seem to be supported in wandb/mlflow integrations, which seem focused on viewing individual trial results without reusing them.
4. How to make trials reproducible? To reproduce a specific trial, it seems like it's necessary to separate the optuna code from the model training code so that it can be run independently. So far, our suggestions for doing hyperparameter tuning involve making dvc exp run calls for every trial, which ensures reproducibility, but it's clunky to implement. Is there a way to support DVC-like reproducibility in a callback or something similarly easy for users to implement?

[dberenbaum]

Revisiting this after talking with @daavoo and @tibor-mach

1. Still TBD.
2. Completed.
3. I would say it's nice to have and related to log_model #305.
4. Let's not worry about this for now. It's enough to track and retrieve the trial results.