Add color/channel support

[billbrod]

Variety of stuff:

• look for SSIM and color references
• make sure synthesis methods can take and return multi-channel images (though what they do should depend on the model)
• Integration with colour?

[billbrod]

Create a notebook describing what to do with color? Show the texture model (and one more?), and how methods that operate on channels independently can reasonably handle color if you transform the inputs from RGB to something else. Can do this as either a preprocessing step or as part of the model, pluses and minuses to both

[billbrod]

Playing around with the Portilla-Simoncelli texture model, I got the best behavior by doing the following (color transform happens within forward method):

• Changing the loss: compute L2 loss separately on each channel, then use torch.logsumexp to combine them (helps keep the error in the separate color channels more aligned)
• Using the rgb2ocp transform from Brown 2023, as seen in this code, which transforms RGB to an LMS-like space, then from that into opposing color channels. (define rgb2opc as in that code, then transform image in model with opc_image = einops.einsum(image, rgb2opc, 'b c1 h w, c2 c1 -> b c2 h w')). (Random orthogonal projection also works reasonably well, though not as well)
• Similarly, adding Brown 2023's cross-channel correlations.
• Adding marginal stats on RGB image (need to double-check whether this actually helps very much).

Colors don't look as good as in the Brown paper, maybe dig into it a bit to see if there's anything else I'm missing? Though the Brown paper is using the pooled texture metamer, so maybe that helps.

Eero's original color texture code also has a step that manually imposes the global correlations between RGB channels, but I'm not sure this helps that much.

[billbrod]

Vasha DuTell pointed out that in the Brown paper, they ran the synthesis for a long time, that the colors were the last thing to come together. So maybe just let it run longer?

[hmd101]

Notable Differences between Brown and Portilla-Simoncelli (PS)-Method

0. General note:

• They don't extend the SP-Method directly to introduce the pooling feature aiming at pooling over multiple smaller regions in the image due to computational costs (ca. six hours for the synthesis of one 512x512 image).
• Instead, they use a data flow network with the desired statistics embedded.

1. Regarding Image Preprocessing
   Brown et al. …

• for color images:
  1. split image into color channels
  2. transform RGB channels to opponent color space, which consists of the following channels: achromatic, red-green, and blue-yellow (see code)
• for gray images:
  • only use achromatic channel of opponent color space

2. Regarding Statistics
   Brown et al. …
   • … use raw (rather than central) statistics (i.e., they omit the mean subtraction)
     • and also match the mean values
       • PS exclude them. For example of PS-method with magnitude means included, refer to this notebook.
   • … removed some statistics (as it did not seem to affect metamer quality)
     • low-pass component statistic
     • all of the autocorrelation statistics
     • benefit: computational savings
     • As opposed to S-P who pool over the entire image, Brown et al. pool over multiple smaller regions in the image. They hypothesize that this might render the autocorrelations redundant.
   • … add cross-channel correlations for color images
     • between edge magnitudes and phase components at the same scale and orientation
   • … introduce new statistics capturing end-stopping
     • by measuring the change in strength along an edge
       • compute via “subtracting each edge magnitude component image from a copy of itself translated a short distance along the expected edge direction (i.e., the filter orientation) and then squaring the result” (Brown et al, 2023)
       • in their tests, they observed improvements in regions with lines that curve or end.