AttributeError: 'NoneType' object has no attribute 'flatten'. Intel OneAPI optimisation introduces error

[karnikkanojia]

So I'm trying out this Intel OneAPI to utilise hardware optimisation of Intel CPU. I believe they have their own implementations for layers and modules that act as a wrapper around torch modules.

Here is the minimal reproducible code for you to test out:

import torch
import intel_extension_for_pytorch as ipex
import torchxrayvision as xrv
from torchcam.methods import GradCAM
input_tensor = torch.rand((1, 1, 224, 224))
model = xrv.models.DenseNet(weights="densenet121-res224-all")
model = ipex.optimize(model, dtype=torch.float32)
target_layer = model.features.get_submodule("denseblock4.denselayer16.conv2")
cam = GradCAM(model, target_layer)
scores = model(input_tensor)
cam(class_idx=1, scores=scores)

So I believe the _hook_g, backward hooks seems to be returning None. Can you help me in this or maybe understand that I should file a issue with the Intel repository.

Here is the stack trace for the error.

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[2], line 10
      8 input_tensor = torch.randn(1, 1, 224, 224)
      9 scores = model(input_tensor)
---> 10 cam(class_idx=1, scores=scores, )

File /usr/local/lib/python3.10/dist-packages/torchcam/methods/core.py:169, in _CAM.__call__(self, class_idx, scores, normalized, **kwargs)
    166 self._precheck(class_idx, scores)
    168 # Compute CAM
--> 169 return self.compute_cams(class_idx, scores, normalized, **kwargs)

File /usr/local/lib/python3.10/dist-packages/torchcam/methods/core.py:193, in _CAM.compute_cams(self, class_idx, scores, normalized, **kwargs)
    178 \"\"\"Compute the CAM for a specific output class.
    179 
    180 Args:
   (...)
    190         the k-th element of the input batch for class index equal to the k-th element of `class_idx`.
    191 \"\"\"
    192 # Get map weight & unsqueeze it
--> 193 weights = self._get_weights(class_idx, scores, **kwargs)
    195 cams: List[Tensor] = []
    197 with torch.no_grad():

File /usr/local/lib/python3.10/dist-packages/torchcam/methods/gradient.py:105, in GradCAM._get_weights(self, class_idx, scores, **kwargs)
    103 self.hook_g: List[Tensor]  # type: ignore[assignment]
    104 # Global average pool the gradients over spatial dimensions
--> 105 return [grad.flatten(2).mean(-1) for grad in self.hook_g]

File /usr/local/lib/python3.10/dist-packages/torchcam/methods/gradient.py:105, in <listcomp>(.0)
    103 self.hook_g: List[Tensor]  # type: ignore[assignment]
    104 # Global average pool the gradients over spatial dimensions
--> 105 return [grad.flatten(2).mean(-1) for grad in self.hook_g]

AttributeError: 'NoneType' object has no attribute 'flatten'

[frgfm]

Hey @karnikkanojia 👋

I don't know about Intel OneAPI so I'm not sure I can help much. Have you tried commenting the line model = ipex.optimize(model, dtype=torch.float32). If it works when you're commenting that, most likely, ipex changes things related to the pytorch hooks and are not, by default, compatible with torchcam I'm afraid

[karnikkanojia]

Thanks for looking at it 👍🏽. It does work perfectly without it. Wanted to see if I can optimize model processing time with the Intel OneAPI on CPU. I will file an issue on their side and lets see what is the problem there.

[frgfm]

Roger that! If it works without it, my guess is that the optimization does merge/change layers so a few tensor hooks might have been removed. Because what the stack trace says is:

• when performing backprop, no gradient has been collected by the CAM extractor hooks
• so my best guess is that the hook is removed when using ipex

Have you tried to do check whether the ipex optimized model has the exact same layout than the default PyTorch one?

[karnikkanojia]

So I tried to get summary for the architecture to check the changes. It's just they have wrappers over Conv2d and Linear layers. Went through their docs and found that they fuse layers using oneDNN but I don't see any fusion though the layers were same.

[frgfm]

To be honest, I don't really know but what I observe is that without the intel line, this works fine 👌

[karnikkanojia]

Hey, thank you for looking out for the problem. Opened an issue with the maintainers. Here is the link(it's closed though now): issue