Update to support AutoGluon v0.4

[Innixma]

PR to add support for AutoGluon v0.4

• Updates install from source logic to align with v0.4
• Added AutoGluon_hq and AutoGluon_gq presets that represent mid-way points between the quality of AutoGluon_bestquality and the inference speed of AutoGluon.
• Added loading test data prior to inference for a more genuine inference latency (previously loading data from file was included in inference time)
• Added predictor.persist_models('best') call to avoid loading models from disk during inference (for more genuine inference latency estimates).

[PGijsbers]

Why are these dependencies installed separately?

[Innixma]

Currently it isn't part of our build-from-source install by default, and I haven't tested across windows, mac, and ARM to ensure this isn't unstable in those configs. It is stable on Linux though. If it is important, I could switch to having it in the same line as the general pip install (PIP install autogluon.tabular[all,skex]).

[Innixma]

Basically the alternative is to shift it to be specified in two locations:

for pypi install:

instead of:
PIP install autogluon

it would be:

PIP install autogluon.tabular[all,skex]

and for source install:

instead of

PIP install -e tabular/[all]

it would be:

PIP install -e tabular/[all,skex]

[PGijsbers]

I was just trying to consider ways to make the relationship of the installation script to current versions less hard-coded. In principle, we would like to be able to provide backward compatibility to older versions of the frameworks when reasonable.

[Innixma]

This is hard currently because of the existence of PIP being different from pip. I'm unsure the reasoning for this in AutoMLBenchmark, but it is the reason I can't use AutoGluon's default build-from-source logic mentioned in our install instructions: https://auto.gluon.ai/stable/index.html

This is the file I'd like to call to build AG from source: https://github.com/awslabs/autogluon/blob/master/full_install.sh

However, I can't because this file uses pip while I need to use PIP in AutoMLBenchmark.

If this can be resolved then it should be do-able to have backwards compatibility

[sebhrusen]

@Innixma see my suggestion below.

I'm unsure the reasoning for this in AutoMLBenchmark

it was mainly to hide the automatic creation of the venv, but still be able to access both default python exec and the venv python exec if needed.

the exec is stored in $py_exec variable if you need it, for example to set a custom PATH as I suggest below. I think this should be enough for your needs.

[Innixma]

Thanks for this! I wasn't so savvy with bash so this was very helpful.

[Innixma]

Re scikit-intelex, I've resolved this by not hardcoding the version but instead using the version defined via the skex extra_dependencies key.

[PGijsbers]

@sebhrusen it feels like this is getting unwieldy. Should we extend the configuration format to more easily allow for different presets?

[sebhrusen]

well, I don't have the feeling that the configuration format is the problem here (or is it? do you have sth in mind like a specific CLI syntax only for presets? sth that would be simple enough to work for various frameworks?)

I mean users can create as many framework definitions in their user frameworks.yaml, we don't have to accept all of those in the default frameworks.yaml.

[sebhrusen]

currently, CLI syntax allows passing framework params like
-Xf.presets=good_quality but this works only for local runs, this is not forwarded to ec2 instances or to docker images, we need custom frameworks.yaml for that, but maybe this could be improved (whitelisting?) although I'm not sure I like the idea of having params coming from multiple sources when executed remotely.

[PGijsbers]

I don't really know what I had in mind at the time. But if every framework does this, it will blow up the configuration file many times over. MLJar has 4 presets, GAMA has more presets, and I suspect (and hope) more frameworks might follow.

A custom parameter which is specifically for the mode/preset might be reasonable, considering it's a "special" hyperparameter in the sense that it's the only one we allow for benchmarking. Consider the last three additional entries in the framework definition:

AutoGluon:
  version: "stable"
  description: |
    AutoGluon-Tabular: Unlike existing AutoML frameworks that primarily focus on model/hyperparameter selection,
    AutoGluon-Tabular succeeds by ensembling multiple models and stacking them in multiple layers.
  project: https://auto.gluon.ai
  refs: [https://arxiv.org/abs/2003.06505]
  presets: ['best_quality', 'high_quality', 'good_quality']
  preset: best_quality    # could also be the first of `modes`  by convention
  param_name: presets  # the framework name for the hyperparameter, so that we can easily convert internally to add it to `params` to avoid changing integration scripts

This could be complimented by a CLI argument which overrides the mode argument. It wouldn't solve the issue of forwarding.

On the other hand, it requires additional logic for something that works well already (even if it's minimal).

[sebhrusen]

@PGijsbers maybe the first mistake was to allow non-default presets in the resources/frameworks.yaml.
The tool offers the possibility for anyone to add custom configurations, we're not going to limit that, and for the paper's benchmarks, we use the franeworks_yyyyQn.yaml files where we can decide to use non-default presets after agreeing with each team ("was it a good idea to allow multiple configs for those?" is a separate question).
The problem with making preset special is that we're officially encouraging this approach (which I like btw), and, as we already see, it creates issues on our side (multiplication, way to fine tune the framework only for the benchmark, in contradiction with the original approach of benchmarking all tools with their default setup...).

That's why personally, I'd go backwards and accept only default setup in frameworks.yaml, rather than creating a new convoluted syntax. And we would decide "customizations" only for the franeworks_yyyyQn.yaml ones.
My second choice would be to just add a param/section to the framework description whitelisting framework params that are always forwarded, it's relatively simple to do and will just allow the usual CLI syntax also for docker and aws.

[PGijsbers]

That's why personally, I'd go backwards and accept only default setup in frameworks.yaml, rather than creating a new convoluted syntax. And we would decide "customizations" only for the franeworks_yyyyQn.yaml ones.

Agree. Since there are already "duplicate" entries (varying only in mode) in the other files, I suggest we OK it for this PR and refactor them all out at once afterwards.

My second choice would be to just add a param/section to the framework description whitelisting framework params that are always forwarded, it's relatively simple to do and will just allow the usual CLI syntax also for docker and aws.

Let's hold off on that for now, I guess we should put more emphasis on people using their local (user) configuration file to keep variations of the different frameworks/configurations that they are interested in.

[PGijsbers]

Thanks for the update and sorry for the inactivity. I will be defending my dissertation in two weeks, and I've been very busy with all the necessary preparations. Things should return to normal (for me) starting in June.

[Innixma]

No worries @PGijsbers , and thanks for your review :). Hope your dissertation goes well!

[sebhrusen]

what about sth like:

env PATH=`dirname $py_exec`:$PATH  bash -c ./full_install.sh

haven't tried, but shouldn't it work?

[Innixma]

Thanks! This makes things a lot nicer and allows us to support AutoGluon v0.1.0+ all at once. Refer to the updated PR.

[Innixma]

@PGijsbers @sebhrusen thanks for the comments! I have updated the PR to reflect them and this version should now be compatible with AutoGluon versions >=0.1.0 (including v0.4, v0.5, and soon to be released v0.5.1).

[PGijsbers]

I can't test it currently (I can't install fairscale locally and right now have no access to aws), but in principle it looks good to me.

[Innixma]

Any other action on my end to take for this PR?

[PGijsbers]

I was waiting to see if @sebhrusen agrees. We don't (yet) have a policy in place that determines what to do if someone does not check in.

[sebhrusen]

@PGijsbers do we agree to create a follow-up ticket to clean up the frameworks.yaml?