Random Feature development

[odunbar]

Purpose

To create a random feature implementation in Julia as a MachineLearningTool

• Wrapper for Distributions.jl (Wish to sample Univariate distributions, as well as Multivariate Gaussians/Student-t)
• Ability to easily get/set the parameters from such distributions (e.g. mean/std for example), and resample such distributions
• Build Random Fourier Features (i.e. cosine based)
• Build Random Neuron Features (i.e. based on selecting 1D activation functions)
• Fit a 'Random Feature kernel' to data, by solving the relevant LS optimization problem efficiently.
• (Future) Train the hyperparameters of the distributions or regularization (e.g with a derivative free ensemble method)

[odunbar]

Hi @nickhnelsen I have tasked @tsj5 with getting a nice modular RF package together for use in CES.

We will translate from a python repo, but I don't know if you have any julia code written as an entry point already for RF here

[nickhnelsen]

Hi Ollie and Tom, I think our Python repo (private on Ollie's account) is the best entry point because my old Julia code was for vector-valued RFs.
I would also suggest adding to the above ToDo's the ability to sample Multivariate Student's t distribution (not just Multivariate Gaussian). In fact, a student's t sample only requires a single Multivariate Gaussian sample plus a uni-variate Chi Square sample: https://en.wikipedia.org/wiki/Multivariate_t-distribution

[odunbar]

Hi @nickhnelsen , thanks! Yeah I've already added Tom to the python repo as an entry. I wasn't sure if with all the DARPA grant work there was any existing Julia code worth building on.

[odunbar]

Update

RandomFeatures.jl can now perform all of the checklist for ND->1D features(why i have ticked them off)

I'll begin a new branch orad/RF to see if we can come up with a good interface for scalar-valued RF in CES using the (currently unregistered version of the package)

[odunbar]

Update

RandomFeatures.jl now contains;

• methods for ND->1D
• methods for ND -> PD
• Examples detailing the framework of how to learn RF hyperparameters

The CES package #194 takes in both forms (ScalarRandomFeatureInterface and VectorRandomFeatureInterface), provides some good defaults encoding

• Learning a MatrixVariate Gaussian Distribution (requiring an input covariance, output covariance and zero-mean)
• Different complexity levels for learning the input & output covariance (via diagaonalize_input diagonalize_ouput flags)
• Generic default priors for the algorithm to learn the entries of the in/out covariances

[odunbar]

Updated todo's - after the PR #194 is merged :

• interface for passing in hparam algorithm options
• interface for defining user-priors
• SVD truncation for vector RF
• Sample stage for GCM application
• Documentation (beyond API)
• multithreading for RF prediction,
• use of RF predict! in MCMC