gradcam for a model with a module that outputs a tuple of tensors rather than a tensor

[pigen01]

In my model, I have a module like

class QKVLinear(nn.Module):
    def __init__(self, ...):
        ...
        self.qkv = nn.Linear(dim, qk_dim + qk_dim + dim, bias=bias)
        ...
    def forward(x):
        q, k, v = self.qkv(x).split([self.qk_dim, self.qk_dim, self.dim], dim=-1)
        return q, k, v

thus report AttributeError

  File "/home/xxx/anaconda3/envs/torchcam/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1547, in _call_impl
    hook_result = hook(self, args, result)
                  ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/xxx/anaconda3/envs/torchcam/lib/python3.11/site-packages/torchcam/methods/_utils.py", line 34, in _record_output_shape
    output_shapes.append((name, output.shape))
                                ^^^^^^^^^^^^
AttributeError: 'tuple' object has no attribute 'shape'

any methods to avoid this?

[May2333]

Hi, i have the same issue. did you solve it?

[louiseliuming]

Hi 👋, torch-cam provider @frgfm, thanks to your time. I have the same issue. For all the methods like CAM, GradCAM and LayerCAM etc, the program will report this error 'AttributeError: 'tuple' object has no attribute 'shape''. Could you give me some help? Thanks very much.

[frgfm]

Hey everyone 👋

Apologies for the late reply! Long story short: the library was initially designed for highway networks (taking 1 input, returning 1 output). I've modified it over time to accomodate new models.

So actually there are two topics at end here:

• generally speaking if your problem is only that it's incompatible with models that have several outputs, that's not a huge modification and I can come up with a fix
• Now the other part is that CAM methods implemented here were designed for CNN. And for transformer-based architecture, the same methods aren't valid, so in a way, I prefer that the library crash when you try it on a transformer rather than yielding wrong results.

If some of you are willing, I'm happy to chat about improvements that could be made and how you can help as I indeed want all types of Pytorch models to be correctly supported 😄

Take care!

[louiseliuming]

Hi, torch-cam provider @frgfm, maybe rewrite hook functions which contain activation hook and gradient hook can solve this problem.

[frgfm]

That's what it would take yes! If you're willing, I'm more than happy to guide you to open a PR. Otherwise it will have to wait a bit that I get more bandwidth 😅

[louiseliuming]

Hh hi @frgfm, I'm willing to contribute to this repository, but I may not have as much experience as you, so you're welcome to guide me😊