Random Feature-based CES

[odunbar]

Purpose

Adds the ability to use (Scalar and vector-valued) RF with uncertainty in place of GP within CES. using RandomFeatures.jl

Closes #164

Content

• Interfaces with the currently registered RandomFeatures.jl
• adds ScalarRandomFeatureInterface as a MachineLearningTool
• adds VectorRandomFeatureInterface as a MachineLearningTool
• new example examples/Emulator/RandomFeature/optimize_and_plot_RF.jl
• new example examples/Emulator/RandomFeature/vector_optimize_and_plot_RF.jl
• new example examples/Lorenz/calibrate.jl
• new example examples/Lorenz/emulate_sample.jl

The current implementation has Scalar RF replacing (exactly) the GP, whereas Vector RF does no SVD, and therefore learns the output space correlations. The hyperparameter learning is more involved, so to reduce some cost I learn the cholesky factors of an input and output covariance of the feature distribution, currently described by a MatrixVariate Normal distribution.

• new example examples/GCM/emulate_sample_script.jl though currently just the emulation!

In this example we have 4 options: (Note in all cases we train on cholesky factors for the input variables)

1. GPR trains an output_dim-length vector of scalar GPRs,
2. Scalar RFR SVD replaces the vector of scalar GPRs, with a vector of scalar RFRs,
3. Vector RFR SVD Diagonal assumes a diagonalized output in the vector problem (i.e. still in the setting of a system of Scalar RFs & GPs but only train one object)
4. Vector RFR SVD nondiagonal still applies the SVD, but does not assume that the resulting output must be diagonal. It therefore learns cholesky factors of the output
5. Vector RFR nondiagonal does not apply SVD, nor assumes the output is diagonal. It learns the cholesky factors of the direct output.

Emulating an R^2 to R^2 function (150 data points)

1. SVD + Scalar GP (diag in) results
   

2. SVD + Scalar RF (nondiag in) results
   

3. SVD + vector RF (diag out) results
   

4. SVD + vector RF (nondiag out) results
   

5. vector RF (diag out) results
   

6. vector RF (nondiag out) results
   

Emulating GCM data R^2 -> R^96, evaluated at a test point

SVD + Scalar RF results

SVD + Vector RF (restrict to diagonal) results [hparam learnt with 202 features])

SVD + vector RF results (full non-diagonal [hparam learnt with 608 features])

No-SVD, with vector RF results (full non-diagonal) + standardize each data-type by median

SVD + GP results

Full CES test (with "E" emulating an R^2 -> R^12 forward map) 250 data points

1. SVD + Scalar GP (diag in) results
2. SVD + Scalar RF (diag in) results
3. SVD + Scalar RF (nondiag in) results
4. SVD + vector RF (nondiag in, diag out) results
5. SVD + vector RF (nondiag in, nondiag out) results
6. vector RF (nondiag in, diag out) results
7. vector RF (nondiag in, nondiag out) results

[codecov]

Codecov Report

Patch coverage: 5.82% and project coverage change: -38.88 ⚠️

Comparison is base (045ee4a) 88.65% compared to head (1ddc959) 49.78%.

❗ Current head 1ddc959 differs from pull request most recent head c8a993c. Consider uploading reports for the commit c8a993c to get more accurate results

Additional details and impacted files

@@             Coverage Diff             @@
##             main     #194       +/-   ##
===========================================
- Coverage   88.65%   49.78%   -38.88%     
===========================================
  Files           4        6        +2     
  Lines         388      697      +309     
===========================================
+ Hits          344      347        +3     
- Misses         44      350      +306     

Impacted Files	Coverage Δ
src/ScalarRandomFeature.jl	0.00% <0.00%> (ø)
src/VectorRandomFeature.jl	0.00% <0.00%> (ø)
src/Emulator.jl	86.88% <64.28%> (-6.76%)	⬇️
src/MarkovChainMonteCarlo.jl	79.85% <100.00%> (ø)

... and 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[odunbar]

Note: Docs build locally, just not with the online setup - which does not dev the local repo.

[odunbar]

Unfortunately it is only myself developing the repo currently so merging without review.
Some reassurance, (1) this is a largely standalone feature, (2) it has comprehensive testing, (3) it has some comprehensive examples (4) the API and constructors are documented. A new issue #215 has been opened to schedule the future work,

Use of RF seamlessly in this codebase will still need a little work, but at this point the tool is in a good-to-go state with some reasonable defaults. I'm looking forward to future manageable PRs that will develop on this!

[odunbar]

bors r+

[bors]

Build failed:

• docs-build

[odunbar]

bors r+

[bors]

Build failed:

• docs-build

[odunbar]

bors r+

[bors]

Build failed:

• docs-build

[odunbar]

bors r+

[bors]

Build failed:

• test-macos

[odunbar]

bors r+

[bors]

Build succeeded!

The publicly hosted instance of bors-ng is deprecated and will go away soon.

If you want to self-host your own instance, instructions are here.
For more help, visit the forum.

If you want to switch to GitHub's built-in merge queue, visit their help page.

• buildkite/calibrateemulatesample-ci
• docs-build
• test-linux
• test-macos
• test-windows