Data Preprocessing Configuration

[simon376]

Hi,

I'm currently using this template for my project, it's great!
But what I'm struggling with, is trying to add my data preprocessing to the YAML configurations.
I'm using combining torchvision.transforms using transforms.Compose, and I want to define and plug-together the different steps using Hydra. But I'm having a hard time, trying to pull this off, I'm confused about how the "inheritance" works.

I have a datamodule.yaml including

data_dir: XXX
batch_size: 16
train_test_split: (...)

and I have / want a augmentations.yaml:

# instantiate pipeline straight from this file
_target_: torchvision.transforms.transforms.Compose

randomrotation:
  _target_: torchvision.transforms.transforms.RandomRotation
  degrees: ??? # sequence or number

randomhflip:
  _target_: torchvision.transforms.transforms.RandomHorizontalFlip

randomvflip:
  _target_: torchvision.transforms.transforms.RandomVerticalFlip

In the DataModule, I want to compose my preprocessing steps dynamically by experiment, so switching out different parts of the preprocessing/augmentation. but they should ofc be all combined in one Compose object

My approach has been to use the variable interpolation, but that does not really work as intended:
(datamodule.yaml)

defaults:
  - dataprocessing@augmentations: augmentations.yaml
  - dataprocessing@transforms: transforms.yaml

augmentation_pipeline:
  _target_: torchvision.transforms.transforms.Compose
  transforms:
    - ${augmentations.randomrotation}
    - ${augmentations.randomhflip}
    - ${augmentations.randomvflip}

also, I'd like to "globally" set parameters like the randomrotation.degrees, without having to access dataprocessing.augmentations.randomrotation.degrees, but if I put it at top-level at the datamodule file, ofc it tries to use it as a keywordargument for the DataModule instantiation..

any tips? I'm not 100% getting the hang of hydra yet..