ClasswiseWrapper yields different results

[patrickFuerst]

🐛 Bug

Using JaccardIndex with ClasswiseWrapper results in different values than without ClasswiseWrapper.

To Reproduce

Steps to reproduce the behavior...

Run the code snippet.

Code sample

from torchmetrics import ClasswiseWrapper, JaccardIndex
import torch 

target = torch.randint(0, 2, (10, 25, 25))
preds = [ torch.randint(0, 2, (10, 25, 25)) for i in range (3)]
jaccard_single = JaccardIndex(num_classes=2, average=None)
class_wrapper = ClasswiseWrapper(
        JaccardIndex(num_classes=2, average=None),
        labels=["class1", "class2"]
    )

for p in preds:
    print("Metric ",jaccard_single(p,target))    
    print("Wraped metric ",class_wrapper(p,target))

The code produces the following output:

Metric  tensor([0.3351, 0.3333])
Wraped metric  {'jaccardindex_class1': tensor(0.3351), 'jaccardindex_class2': tensor(0.3333)}
Metric  tensor([0.3293, 0.3357])
Wraped metric  {'jaccardindex_class1': tensor(0.3322), 'jaccardindex_class2': tensor(0.3345)}
Metric  tensor([0.3424, 0.3435])
Wraped metric  {'jaccardindex_class1': tensor(0.3356), 'jaccardindex_class2': tensor(0.3375)}

Expected behavior

I would expect that the wrapped metric outputs the same values as the simple JaccardIndex.

Environment

• TorchMetrics version (and how you installed TM, e.g. conda, pip, build from source):
  Installed version 0.9.2 using pip
• Python & PyTorch Version (e.g., 1.0):
  Tested with Python 3.8.14 and pytorch 1.3.1
• Any other relevant information such as OS (e.g., Linux):

Additional context

[github-actions]

Hi! thanks for your contribution!, great first issue!