It seems some bug in split_dataset

[aiboys]

When building dataset dict, transformers would be overloaded in train and val split dataset in split_dataset

I fixed such bug using following codes:

import copy
dataset_dict[sub_dataset_id] = copy.deepcopy(sub_dataset) 

[yoshitomo-matsubara]

Hi @aiboys

Thank you for reporting that!

It seems that Subset class in PyTorch simply shallow-copies the original dataset. In PR #210, I made it deep-copy before the for-loop to fix the issue, instead of when substituting to dataset_dict. Otherwise, other splits might be affected by the change made before dataset_dict[sub_dataset_id] = copy.deepcopy(sub_dataset) you showed above.