More image metrics

[Borda]

🚀 Feature

just found this package and seems we have not yet all Image metrics :]
some inspiration came from https://github.com/andrewekhalel/sewar but we aim on own implementation with torch

Alternatives

• Mean Squared Error (MSE)
• Root Mean Sqaured Error (RMSE)
• Peak Signal-to-Noise Ratio (PSNR)
• Structural Similarity Index (SSIM)
• Universal Quality Image Index (UQI)
• Multi-scale Structural Similarity Index (MS-SSIM)
• Erreur Relative Globale Adimensionnelle de Synthèse (ERGAS)
• Spatial Correlation Coefficient (SCC)
• Relative Average Spectral Error (RASE)
• Spectral Angle Mapper (SAM)
• Spectral Distortion Index (D_lambda)
• Spatial Distortion Index (D_S)
• Quality with No Reference (QNR)
• Visual Information Fidelity (VIF)
• Block Sensitive - Peak Signal-to-Noise Ratio (PSNR-B)

Additional context

any of these metrics would be a separate PR/addition :]

[nishant42491]

i would like to work on this can you assign it to me

[ankitaS11]

Hi, @Borda

Thanks for creating the issue tracker. Would love to help in this, as I also look forward to contributing to the lightning framework.

I would like to begin with MSE and RMSE, if that's fine with you and everyone else. Please let me know if I can start working on these two metrics.

Thanks!

[SkafteNicki]

@ankitaS11 Please note we already have MSE and RMSE in the regression package:
https://torchmetrics.readthedocs.io/en/latest/references/modules.html#meansquarederror (this takes an squared argument)
So I do not think we should re-implement them in the image domain.

[Borda]

i would like to work on this can you assign it to me

cool, which one do you want to take? (we have many ⚡)

[ankitaS11]

@ankitaS11 Please note we already have MSE and RMSE in the regression package: https://torchmetrics.readthedocs.io/en/latest/references/modules.html#meansquarederror (this takes an squared argument) So I do not think we should re-implement them in the image domain.

Hi, @SkafteNicki - Got it. Thank you for updating the issue as well, I can try Quality with No Reference then, just want to also make sure that it doesn't collide with @nishant42491. Nishant, curious to know if you have any metric in mind to pick up first? I can pick the others then, if that's okay.

[justusschock]

@Borda @SkafteNicki are we aiming for an integration or a direct torch reimplementation here?

[Borda]

@Borda @SkafteNicki are we aiming for an integration or a direct torch reimplementation here?

reimplementation, I put this package just as inspiration for what other image metrics exist...

[justusschock]

reimplementation, I put this package just as inspiration for what other image metrics exist...

I'd think so as well :)

@ankitaS11 @nishant42491 You can however use this package for testing :)

[nishant42491]

@Borda @ankitaS11 may I start with implementing spatial correlation coefficient first?

[justusschock]

@nishant42491 sure, I'll add you to it. Open a draft PR early and ping us there if you need help :)

[nishant42491]

@nishant42491 sure, I'll add you to it. Open a draft PR early and ping us there if you need help :)

thank you :)

[Piyush-97]

I would like to work on Relative Average Spectrul Error, it would be a great way for me to learn new stuff.

[ankitaS11]

Hi, @Borda

I was going through the referenced repo (sewar) for QNR, and found that it depends on: D_LAMBDA and D_S - and both of them depend on UQI. So, probably it will be easier and more sensible to implement UQI first, and then go to D_LAMBDA, D_S and finally QNR.

Do you have any opinions on this? Just wanted to confirm if I'm moving in a right direction here. In a gist, I would like to implement UQI first, and then other metrics if that's okay with you and everyone.

[justusschock]

Hey @ankitaS11 in that case it definitely makes sense to go with your proposed way :) I updated the issue. Just let us know here, when I should assign new metrics to you :)

[ankitaS11]

Hey @ankitaS11 in that case it definitely makes sense to go with your proposed way :) I updated the issue. Just let us know here, when I should assign new metrics to you :)

Hi @justusschock, I am done with the implementation of UQI metric, now I would like to work on D_Lambda.

[vumichien]

@Borda I will work on Spectral Angle Mapper (SAM)

[Paul-Aime]

@vumichien if I'm not mistaken, SAM is just the arccos of the cosine similarity which is implemented here.

[Paul-Aime]

@vumichien My bad, SAM being an image metric and being about spectral characteristics, it is needed to assume NCHW shape, and using torchmetrics.functional.cosine_similarity with those assumptions will require some useless reshaping.

Here is some code as an excuse and to explain myself:

Code

import torch
import torchmetrics
import einops


SPECTRAL_DIM = 1
N, C, H, W = 13, 5, 32, 64


def main():

    x = torch.randn(N, C, H, W)
    y = torch.randn(N, C, H, W)

    sam_index1 = sam_from_torchmetrics_cossim(x, y, reduction="mean")
    sam_index2 = sam_from_scratch(x, y, reduction="mean")

    assert torch.allclose(sam_index1, sam_index2)


def sam_from_torchmetrics_cossim(pred, target, *, reduction="mean"):
    # (..., N, d) required for `torchmetrics.functional.cosine_similarity`
    pred_ = einops.rearrange(pred, "n c h w -> n (h w) c")
    target_ = einops.rearrange(target, "n c h w -> n (h w) c")
    cossim = torchmetrics.functional.cosine_similarity(pred_, target_, reduction="none")
    return reduce(cossim.arccos(), reduction=reduction)


def sam_from_scratch(pred, target, *, reduction="mean"):
    return reduce(_angle(pred, target, dim=SPECTRAL_DIM), reduction=reduction)


def _angle(x1, x2, *, dim):
    return _cosine_similarity(x1, x2, dim=dim).acos()


def _cosine_similarity(x1, x2, *, dim):
    x1 = x1 / torch.norm(x1, p=2, dim=dim, keepdim=True)
    x2 = x2 / torch.norm(x2, p=2, dim=dim, keepdim=True)
    return (x1 * x2).sum(dim=dim).clamp(-1.0, 1.0)


def reduce(x, reduction):
    if reduction == "mean":
        return x.mean()
    if reduction == "sum":
        return x.sum()
    if reduction is None or reduction == "none":
        return x
    raise ValueError(
        f"Expected reduction to be one of `['mean', 'sum', None]` but got {reduction}"
    )


if __name__ == "__main__":
    main()

[soma2000-lang]

@Borda It would be great if you can give me some reference regarding Block Sensitive - Peak Signal-to-Noise Ratio (PSNR-B).
Then I would like to work on adding it as a metrics.
Thanks!

[Borda]

@SkafteNicki could you pls link some good resources? :)

[rschireman]

I can take a stab at Visual Information Fidelity VIF if it hasn't been taken yet @Borda

[SkafteNicki]

@Borda It would be great if you can give me some reference regarding Block Sensitive - Peak Signal-to-Noise Ratio (PSNR-B). Then I would like to work on adding it as a metrics. Thanks!

Hi @soma2000-lang,
Here is the reference paper:
https://ieeexplore.ieee.org/abstract/document/5535179
and here is a reference implementation in numpy:
https://github.com/andrewekhalel/sewar/blob/ac76e7bc75732fde40bb0d3908f4b6863400cc27/sewar/full_ref.py#L371-L395

[theja-vanka]

Hi @Borda, please assign any metric that needs to be implemented to me. I see a lot of people taking a stab at most of them, so any metric which doesn't conflict with others would be great.
Looking forward to hearing back from you.

[yashika-git]

@Borda, Can I add PSNR-B, if someone isn't already working on it?

[soma2000-lang]

@yashika-git I have started working on it

[Borda]

@yashika-git I have started working on it

Great! @soma2000-lang, updated the assignment for you

@Borda, Can I add PSNR-B, if someone isn't already working on it?
Hi @Borda, please assign any metric that needs to be implemented to me.

@yashika-git @theja-vanka how about QNR or VIT?

[yashika-git]

@Borda, I'd like to implement Visual Information Fidelity (VIF).

[yashika-git]

@Borda, should Intersection over Union (IoU) and Dice Score image metric be added as well?
For reference - MONAI has both of these metrics (https://github.com/Project-MONAI/MONAI/tree/dev/monai/metrics).

[Borda]

@Borda, should Intersection over Union (IoU) and Dice Score image metric be added as well?

yeah, I think we started it somewhere but can't find it now... in such case let's add it :)

[rddunphy]

Would you accept a PR for one not on the list? I have an implementation of complex wavelet structural similarity (CW-SSIM) that I could contribute. (https://ieeexplore.ieee.org/abstract/document/5109651)

[SkafteNicki]

Hi @rddunphy ,
You are more than welcome to send a PR with metrics that are not on the list :)

[bhack]

F-score and Boundary-IOU would be nice to have. See #1500

[Borda]

F-score and Boundary-IOU would be nice to have. See #1500

Would you be interested in adding it? :)

[bhack]

Would you be interested in adding it? :)

I am not familiar with the internals/standard of the library but at least I've posted in the ticket a gist to reproduce the mask to boundary util:
#1500 (comment)

Then I think that you could use your already available internals F and IOU utils to quickly impl the metrics.

[Borda]

I am not familiar with the internals/standard of the library but at least I've posted in the ticket a gist to reproduce the mask to boundary util:

letss have tried, open PR with a draft with a functional version and then we can help to finish it :)

[bojobo]

@Borda, I'd like to implement Visual Information Fidelity (VIF).

@yashika-git How is it going with VIF? Since I currently use the piq implementation and therefore would like a torchmetrics implementation, I can take over.

[yashika-git]

@bojobo, please feel free to takeover :)

[HoseinAkbarzadeh]

Hi @Borda , I was browsing issues and found this topic. I think I can add Spatial Correlation Coefficient (SCC) metric to torchmetrics if it is still needed. I am seeing that previous merge was not successful. I was wondering do you think I can try it ?

[Borda]

@HoseinAkbarzadeh that would be great!

[SkafteNicki]

Closing issue as all metrics have been implemented.
Thanks to all the contributors for your help :)