HyperParameterOptimizer runs only as single task at a time

[navotoz]

Hello

I started the optimizer using the code from the example (with minor changes):

an_optimizer = HyperParameterOptimizer(
    base_task_id=args['template_task_id'],
    hyper_parameters=[
        DiscreteParameterRange('generator/A', values=[0, 1, 2]),
        DiscreteParameterRange('generator/B', values=[0, 1, 2]),
        DiscreteParameterRange('losses/l1', values=[1, 10, 0]),
    ],
    objective_metric_title='mse',
    objective_metric_series='Average',
    objective_metric_sign='min',
    max_number_of_concurrent_tasks=5,
    optimizer_class=aSearchStrategy,
    execution_queue=execution_queue,
    spawn_project=None,  # 'HPO spawn project',
    save_top_k_tasks_only=None,
    time_limit_per_job=None,
    pool_period_min=5,
    total_max_jobs=None,
    min_iteration_per_job=8,
    max_iteration_per_job=None,
    job_complete_callback=job_complete_callback
)

This setup worked fine, until I upgraded the the agent and the clearml pkg to lateset itself yesterdey.

Since yesterday, the optimizer spawns a single new task, only when the old task is done. Thus, only a single worker (out of 3) is working, while the rest are idle.

The version of clearml==1.3.1

Thanks for your help.

[erezalg]

Hi @navotoz, What optimizer are you using?

[navotoz]

Hi @erezalg, aSearchStrategy=OptimizerOptuna.
Everything else in the code is taken straight from the example on your site.

[erezalg]

Hi @navotoz
I might know what it is, we changed the mechanism in which we do job parallelization in optuna per optuna's changes. Let me test it

[navotoz]

Hi @erezalg,
Thanks for that.

[erezalg]

Hi @navotoz, we released a new RC, please check if it solves the issue.
To install, run pip install clearml==1.3.2rc1

[navotoz]

Working as expected!
Thank you.