How to append individual callbacks to a default set of them?

[mariomeissner]

This is a pure Hydra question, but I thought I'd ask here since I assume this issue is common.

Say we have a default config file for callbacks, containing a bunch of them. But I make a new one, and I want to use it sometimes. How can I append it to the defaults from the command line?

Currently, what I do is create a new file defaults_mod.yaml where I copy-paste all the defaults and add the new callback, but that is obviously a terrible approach.

[ashleve]

@mariomeissner As far as I'm aware, appending to yaml list or to config defaults list from command line is currently not supported by hydra in any way.

You can read more here:
facebookresearch/hydra#1547
facebookresearch/hydra#2027
facebookresearch/hydra#1939

Because of that, currently I don't see any approach that would make attaching callbacks more convenient.

This might change in the future hydra releases though.

So the way I usually compose callback configs is either by inheriting defaults:

# in configs/callbacks/my_callback.yaml

defaults:
  - default.yaml # attach default callbacks from configs/callbacks/default.yaml

my_callback: # enable callback from command line with `python train.py callbacks=my_callback`
    _target_: ...
    ...

Or by explicitly storing each callback in submodule:

# in configs/callbacks/default.yaml

defaults:
  - model_ckpt: default.yaml
  - early_stopping: default.yaml
  - my_callback: null # can be enabled from command line with `python train.py callbacks.my_callback=default`

Both solutions are not very elegant but I don't see any better option until hydra releases a more convenient syntax for this task.

[ChristianHinge]

Another way is to join the default callbacks with the callbacks you wish to append using the override syntax. For instance, in one of my experiment yaml files I have:

defaults:
  ...
  - override /callbacks: [default.yaml, freeze_unfreeze.yaml]
  ...

This appends the free_unfreeze.yaml callback to the default callbacks. This way the freeze_unfreeze.yaml callback does not have to inherit default.yaml.