Using SingleTaskMultiFidelityGP with multiple fidelity dimensions for MFBO

[AlexanderMouton]

Hi, I'm doing research where I have a target noisy system that I am optimising with parameters such as allocated time, number of threads and a normalised Hamming distance that measures similarities between different strategies in the noisy system. All of these parameters can be considered as fidelity dimensions, e.g. if I want to optimise the system with 10 seconds and 8 threads allocated while using Strategy A, then (5 seconds, 4 threads and Strategy B with a 0.6 Hamming similarity score) is a low fidelity approximation of the high fidelity system.

The SingleTaskMultiFidelityGP (https://botorch.org/api/_modules/botorch/models/gp_regression_fidelity.html#SingleTaskMultiFidelityGP) allows for a single dimension of fidelity, but not for multiple dimensions. Is there a way to use this for multiple levels of fidelity, or an alternative strategy for this problem?

Thanks!

Alex

[Balandat]

Yeah, though I don't think we have the exact fit for this out of the box. A very basic solution could also be to use a standard SingleTaskGP and just consider two dimensions as the fidelities - though in that case you wouldn't be including inductive biases about the fidelities behaving in a particular way (e.g. via the exponential decay kernel that the SingleTaskMultiFidelityGP uses). You could write your own model by mimicking the SingleTaskMultiFidelityGP and define a kernel for both fidelity values.

[AlexanderMouton]

Hi Balandat,

I've looked into the implementation of SingleTaskMultiFidelityGP and have created a class that mimics it and allows for multiple dimensions by creating an ExponentialDecayKernel/DownsamplingKernel for each of a list of iteration/data fidelities and passing all of them into a ProductKernel and a ScaleKernel thereafter.
I'm still doing some testing on it to see how well it works and will post the code here once I'm sure everything is working properly. Is this something that could warrant creating a PR in your opinion, or is the use case too niche?

Thanks!

Alex

[Balandat]

Great!

I'm still doing some testing on it to see how well it works and will post the code here once I'm sure everything is working properly. Is this something that could warrant creating a PR in your opinion, or is the use case too niche?

Hmm so if this is pretty clean and a strict generalization of the SingleTaskMultiFidelityGP model, then I'd love to see a PR for this.

[AlexanderMouton]

Cool, I'll see if I can make a PR for it tomorrow :)

Thanks!

[AlexanderMouton]

Hi @Balandat

I have successfully modified the SingleTaskMultiFidelityGP and FixedNoiseMultiFidelityGP classes to allow for multiple dimensions of data fidelity with linear_truncated=False, but have run into issues when linear_truncated=True.

In test_gp_regression_fidelity.test_gp(), the following code snippet throws assertion errors:

for p in params:
    self.assertEqual(
        params[p].numel(), m * torch.tensor(batch_shape).prod().item()
    )

With the variables having values:

p = covar_module.base_kernel.kernels.0.covar_module_unbiased.raw_lengthscale
params[p] = Parameter containing:
tensor([[-0.1466, -0.1333, -0.1441]], requires_grad=True)
params[p].numel() = 3
m = 1
batch_shape = torch.Size([])
torch.tensor(batch_shape).prod() = 1.0
torch.tensor(batch_shape).prod().item() = 1.0

I'm finding it difficult to determine whether this is an issue or a side-effect introduced by the added dimensions. In this case I tested it with 3 data_fidelity dimensions, which corresponds to params[p].numel() being equal to 3. Do you possibly have more context on this?

Thanks!

Alex

[AlexanderMouton]

Hi @Balandat

I've added results from a toy problem when using mutliple data-fidelity dimensions (with linear_truncated=True) here

This was for a problem with one problem dimension and two data fidelity dimensions, each with two discrete fidelity values - either 0.75 or 1.0.

g11 has fidelities [1.0, 1.0], g10 has fidelities [1.0, 0.75], g01 has fidelities [0.75, 1.0] and g00 has fidelities [0.75, 0.75].