Add empirical sampling mode to the RCF model

[calebrob6]

The MOSAIKS paper describes RCF, which we've already implemented, and this method (which I'll call MOSAIKS). This method is the same setup as RCF but instead of initializing the conv kernels randomly, you sample patches from the dataset, do ZCA whitening, and use those as the kernels. In some little experiments on EuroSAT and UCM this gives +2 to 5% accuracy over simple RCF kernels.

[isaaccorley]

Just need to add to docs

[calebrob6]

Lol need to figure out some type hints and stuff too

[calebrob6]

probably tests too huh?

[isaaccorley]

Nah who needs tests???

[calebrob6]

Okay nowwww it LGTM

[adamjstewart]

Will review post paper

[estherrolf]

chiming in at Caleb's request on the naming of the option with Gaussian patches and the option with patches drawn from the dataset:

I would call both of these random convolutional features, but in the first the random patches are drawn from a Gaussian distribution and in the second from the empirical distribution of image patches. Thus, I would propose "RCF - Gaussian" and "RCF - Empirical" to distinguish between them, noting that the latter (RCF - empirical) is what what we use in the MOSAIKS paper.

To me the most straightforward would be to have one RCF class, with argument for patch_distribution, where that would be either empirical or Gaussian. It's possible that would be harder to implement, so if it's better to have two different classes, maybe RCF_empirical and RCF_gaussian?

LMK if I can help clarify further!

[isaaccorley]

Would RCF-ZCA or RCF-Empirical-ZCA be better since it's more explicit? I'm assuming we could probably use other techniques than ZCA to estimate the kernels using the dataset.

[estherrolf]

Yeah using ZCA to whiten the images is an optional preprocessing step but we've found that it can help quite a bit. I still think the most straightforward would be to have one RCF class with arguments for patch_distribution as either "gaussian" or "empirical" and have the ZCA whitening step as an optional (but default?) processing step. Alternatively having two classes (RCF_Gaussian) and (RCF_empirical) and for both of them having ZCA whitening step as an argument make sense.

[adamjstewart]

Since we added RCF first, all files and documentation are named after RCF. I haven't read the paper, but my only question is whether we should rename things to be under the RCF or MOSAIKS umbrella.

[calebrob6]

@adamjstewart @estherrolf -- I combined the functionality of both under a single RCF class that takes mode="empirical" or mode="gaussian". I actually like this way better than having two classes and, as usual, am thankful we do thorough peer review :). In particular, I can imagine us adding a few more parameters to this class in the future to allow for easy experimentation over dimensions like "normalization schemes". Having the code in one class is better for this.

[adamjstewart]

I'm fine with the new organization