Allowing for batch processing/multi-processing

[hasoweh]

🚀 Feature

Currently when I use this package, I can only feed a single image through the network at a time in order to process a CAM. However, it would be greatly beneficial to me if I could pass an entire batch of images through the network and obtain a batch of CAMs. This would greatly speed up my processing time, as I am using the CAMs as supervision for a segmentation branch, and at the moment am building a batch of CAMs one at a time (very slow).

I could possibly look into how this could be implemented myself, but since you are more familiar with your own code then maybe you already have an idea of how this could be implemented.

[frgfm]

Hi @SevenFoldDeep 👋

Thanks for the suggestion! Just to make sure, you are referring to generate a batch of CAMs with the same output class index?

If you have a small snippet of code sample you have in mind about what you'd like, that would be great!

[hasoweh]

Hey, thanks for the response.

I thought it would be nice to be able to pass a batch of images to the network, and from the resulting output, get the CAMs for each of the images in the batch, regardless of the output class index. I guess the class indices could be given as a list or tensor such that each element in the list corresponds to one image in the batch.

Here is a code snippet of how that work flow would look like:

# create some CNN model
from torchvision.models import resnet18
model = resnet18()

# create some generator object from a dataset
test = dataloader.Dataset()
generator = torch.utils.data.DataLoader(test)

# define a CAM function
cam_extractor = XGradCAM(model, target_layer = 'layer4', input_shape = (4, 300, 300))

# loop through a batch of images and get their CAMs
for img_batch, labels_batch in generator:
    img_batch = img_batch.to(device)
    # get the logits from the network
    # out has shape = [batch_size, n_classes]
    out = model(img_batch.float())
    
    # get the predicted indices for the entire batch
    # this would give a 1-D tensor of shape = [batch_size]
    preds = [torch.where(F.sigmoid(o).flatten() > threshold)[0] for o in out]
    preds = torch.cat(preds)

    cam_batch = cam_extractor(preds, out)

I guess this would work easiest in a single label classification problem, but maybe it's also possible to have it work for multi-label classification, and then the preds would look something like this

# preds with single label classification
preds = [0, 2, 4, 2, 2, 3, 1, 0, 0, 2] # one class prediction per image
# preds with multi-label classification
preds = [[0, 1], [0, 2], [1], [2], [0, 1, 2]] # 1 or more class predictions per image

Maybe you have some better ideas though how the multi-label could be programmed.

[frgfm]

Thanks for the detailed explanation!

So there are two things to consider here:

• processing a whole batch and computing the CAM of all samples for a single class index: this part is doable and indeed much better than going through all samples manually
• doing the same but with a specific class index for each sample: this part doesn't sound very doable to me. I mean, I can modify the API to do this, but under the hood it wouldn't be any better than a bare for loop, for CAMs that requires gradient information. The reason behind this is that when we backprop to get the gradient, what we backpropagate is class-specific. If it wasn't, the gradient we would get wouldn't be specific to this class.

But for the first part, I can try coming up with something that would work 👌

Let me know what you think!

[frgfm]

@SevenFoldDeep any thoughts? :)

[frgfm]

Moving this to discussion as it's not an issue per-se!

[frgfm]

Hi @SevenFoldDeep 👋

FYI, I just merged #143 which should be of interest to you ;)