Plenoptic model question

[pixelsandpointers]

Hi all,
I am wondering if the plenoptic model needs parameters, or if I can create a torch.nn.Module to just compute RMS contrast (response) of the input image, and later synthesize metamers based on the RMS contrast?

How would I do that when my model only computes something without having parameters?

Example:

class RMS(torch.nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, img):
        mean_intensity = torch.mean(img)
        rms = torch.sqrt(torch.mean((img - mean_intensity) ** 2))
        return rms

Thanks in advance!
Ben

[billbrod]

Thanks for your interest in plenoptic!

Yup, any torch.nn.Module will work with plenoptic. This is how I used it in my foveated metamer project and how most of the models included in plenoptic are implemented. In fact, it's preferred that you don't have any parameters, since the idea is that plenoptic is synthesizing novel stimuli while leaving models unchanged (and so computing the extra gradients is a waste of computation).

However, plenoptic does require that models accept 4d tensors and return 3d or 4d outputs. Your current implementation returns a single scalar, so you'd need to make the following modifications:

class RMS(torch.nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, img):
        mean_intensity = torch.mean(img, dim=[-2, -1]).unsqueeze(-1).unsqueeze(-1)
        rms = torch.sqrt(torch.mean((img - mean_intensity) ** 2, dim=[-2, -1]))
        return rms.unsqueeze(-1)

The two unsqueeze() calls on mean_intensity are only necessary if you want to work with multi-channel inputs (e.g., RGB images), or multi-batch inputs, but are unnecessary if you're operating on only a single grayscale image at a time, as is the case in most of our documentation. Note that the above will compute RMS separately for each channel/batch separately; I'm not sure the proper way to compute RMS across color channels.

I tried to address this point in the docs, did you see my description there?

(Note: I changed this to a discussion, because we want to keep usage questions here, saving issues for bug reports and planning changes to the code base.)

[billbrod]

@benjaminbeilharz Let me know if this works and if you have any more questions!

[pixelsandpointers]

Hi this already helps! Thanks a lot. I might come back to this thread for other questions. 😅