Multistep training with batch_size >=1 per GPU

[jsschreck]

No description provided.

[dkimpara]

can you add some details? is this going to replace all trainers/datasets or just the multistep ones?

[jsschreck]

can you add some details? is this going to replace all trainers/datasets or just the multistep ones?

I just added the single-step dataset. I am aiming for consolidation down to a single trainer with config options to specify what you need to do (single or multi). Working on that and should have it ready by end of week. So far the train_multistep.py in this PR has been tested with the new multi-step datasets. I will update documentation and what not once its all working.

[jsschreck]

@kanz76 I think the bug involving batch size and history len is now corrected.

[kanz76]

I tested

• era5_multistep_batcher.py (exceptMultiprocessingBatcherPrefetch)
• load_dataset_and_dataloader.py
• train_universal.py

They all look good!