Chaining loaded pickle files with transforms

[PCaesar18]

Hi there!

I have been struggling for a while on something which seems quite basic but which I haven't been able to figure out so far.

I have a list of pickled files in multiple directories. Each pickled file consists of a dictionary with a 2 nparray(512x512) items (CT and RT).

Now I want to apply various transformations on just only the CT nparray in the dict.

A simplified version of what I am trying to achieve below:

files = glob.glob(os.path.join(GAN_input_data_folder_name,'**/*.pkl'),recursive = True)

pickleFile = open(files[0], 'rb')
data = pickle.load(pickleFile)

CT_scan = data["CT"]

transform = Compose([
    LoadImage(),
    AddChannel(),
    Resize( spatial_size=[64, 64]),
    ToTensor(),
])
test_data = CacheDataset(CT_scan, transform)

Now I was wondering whether this was possible or whether I would first have to save the CT nparray again as an image.

I hope you are able to help me out. Great project by the way!
All the best,
Caesar

[PCaesar18]

Something that I have been looking at and which I reckon would be what I am looking for is:

class LoadTarJpeg(Transform):
    def __call__(self, data):
        return plt.imread(data)

from the GAN_workflow_array tutorial. But I haven't been able to figure it out for my specific use case yet. Again, any help is appreciated

[Nic-Ma]

Hi @PCaesar18 ,

May I know what's your error message when you try to modify from LoadTarJpeg? Maybe just change to:

files = glob.glob(os.path.join(GAN_input_data_folder_name,'**/*.pkl'),recursive = True)

class LoadPickleD(Transform):
    def __call__(self, data):
       pickleFile = open(data, 'rb')
       return pickle.load(pickleFile)

transform = Compose([
    LoadPickleD(),
    AddChannelD(keys="CT"),
    ResizeD(keys="CT", spatial_size=[64, 64]),
    ToTensorD(keys="CT"),
])
test_data = CacheDataset(files, transform)

Thanks.

[PCaesar18]

Hi Nic,

I used your suggestion and tweaked it a bit further to make it work like a charm.

  class LoadPickleD(Transform):
        def __call__(self, data):
            with open(data, "rb") as file: 
                list_of_dict = pickle.load(file)
            return list_of_dict 

I used to open with "with" to prevent IObuffer from interupting loading.

Thanks for you suggestions!
Best,
Caesar

[Nic-Ma]

Sounds good!