Multi-fidelity BO tutorial doesn't work?

[samuelkim16]

I am going off of the multi-fidelity BO tutorial, which compares multi-fidelity knowledge gradient (KG) with the traditional expected improvement (EI) on just the highest fidelity. At the end, I plot the optimization curves as a function of cost (in the case of multi-fidelity BO, only tracking the highest-fidelity evaluations for the best value so far), but it seems that single-fidelity EI outperforms multi-fidelity KG. Is this not the right way to track this?

fixed_cost = 5.0

cost_ei = []
best_y_ei = []

cumulative_cost = 0

for xi, yi in zip(train_x_ei, train_obj_ei):
    yi = yi[0].item()
    costi = xi[-1].item()
    cumulative_cost += costi + fixed_cost

    if costi == 1:
        cost_ei.append(cumulative_cost)

        if not best_y_ei or yi > best_y_ei[-1]:
            best_y_ei.append(yi)
        else:
            best_y_ei.append(best_y_ei[-1])



cost_mf = []
best_y_mf = []

cumulative_cost = 0

for xi, yi in zip(train_x, train_obj):
    yi = yi[0].item()
    costi = xi[-1].item()
    cumulative_cost += costi + fixed_cost

    if costi == 1:
        cost_mf.append(cumulative_cost)

        if not best_y_mf or yi > best_y_mf[-1]:
            best_y_mf.append(yi)
        else:
            best_y_mf.append(best_y_mf[-1])

import matplotlib.pyplot as plt

plt.figure()
plt.plot(cost_ei, best_y_ei, marker='o', label="Single-fidelity, EI")
plt.plot(cost_mf, best_y_mf, marker='o', label="Multi-fidelity, KG")
plt.xlabel("Cost")
plt.ylabel("Objective")
plt.legend()

Edit:

I just looked at the output of an earlier cell in the notebook where it makes a final recommendation from the multi-fidelity KG, and the output is the following:

recommended point:
tensor([[0.315, 0.138, 0.348, 0.273, 0.300, 0.617, 1.000]],
       dtype=torch.float64)

objective value:
tensor([3.048], dtype=torch.float64)

total cost: 136.0

The objective value is better than it had found during optimization or what EI was able to find. So somehow the GP "knows" about the best objective, but never actually evaluates it? Am I interpreting this correctly?

[Balandat]

The objective value is better than it had found during optimization or what EI was able to find. So somehow the GP "knows" about the best objective, but never actually evaluates it? Am I interpreting this correctly?

Yes, this is correct. MFKG avoids evaluating the high fidelities due to the high cost. So your analysis about how well it does needs to be based on the points it would recommend at each step (not what it evaluates as part of the optimization). Take a look at the ["Make a final recommendation¶"](https://botorch.org/tutorials/discrete_multi_fidelity_bo#Make-a-final-recommendation) section of the tutorial. You'll want to call get_recommendation() in each step based on the model fit on the data observed up to that point. You can compare performance of the two approaches by comparing that recommendation with the value chosen by EI.

This is often referred to as considering "inference regret" rather than "simple regret" - the former is based on what the model believes is best, the latter is simply based on the points that have been evaluated so far.

[samuelkim16]

Great, thank you! So I'm guessing that the literature typically reports these curves based on the recommended points? Is there a "best practices" way to do this for actual expensive functions? Perhaps evaluate the recommended point at regular intervals (e.g. every 10 steps)?

[Balandat]

So I'm guessing that the literature typically reports these curves based on the recommended points? Is there a "best practices" way to do this for actual expensive functions? Perhaps evaluate the recommended point at regular intervals (e.g. every 10 steps)?

Depends. Often papers will evaluate this on tabular or surrogate benchmark problems where the evaluation at different fidelities is actually not that expensive (so that it allows for proper benchmarking). If you're working on an actual expensive black box function you could do as you said and only check that sporadically, or you just run the optimization and then pick the best, provided the fits of your surrogate model are reasonable (can check that via cross-validation).

[samuelkim16]

Reviving this discussion, I am comparing multi-fidelity (MF) BO with single-fidelity (SF) BO, and averaging over 10 trials, am not finding that MF-BO performs any better than SF-BO. Below is a plot of the two optimization curves, with the shaded regions representing +/- 1 standard deviation over the 10 trials. This seems to suggest that the results in the tutorial are cherry-picked? Are there any tips/tricks to improving MF-BO performance?

Most of my code is the same as the discrete MF-BO tutorial, with some changes to track the performance over trials:

# Perform multi-fidelity BO
train_x_list = []
train_obj_list = []
train_rec_list = []
cost_rec_list = []

for i in range(10):
    init_x, init_obj = generate_initial_data(n=10)
    train_x, train_obj = init_x, init_obj
    # print((train_x.shape, train_obj.shape))
    # # > (torch.Size([16, 7]), torch.Size([16, 1]))

    cumulative_cost = 0.0

    cost_init = cost_model(train_x).sum().item()
    cost_rec = []
    train_rec = torch.empty(0, **tkwargs)

    while cumulative_cost < 150:
        mll, model = initialize_model(train_x, train_obj)
        fit_gpytorch_mll(mll)
        mfkg_acqf = get_mfkg(model)
        new_x, new_obj, cost = optimize_mfkg_and_get_observation(mfkg_acqf)
        train_x = torch.cat([train_x, new_x])
        train_obj = torch.cat([train_obj, new_obj])
        cumulative_cost += cost
        # Track inference regret for comparison
        _, new_obj = get_recommendation(model)
        cost_rec.append(cost_init + cumulative_cost)
        train_rec = torch.cat([train_rec, new_obj])
    
    train_x_list.append(train_x)
    train_obj_list.append(train_obj)
    train_rec_list.append(train_rec)
    cost_rec_list.append(cost_rec)

# Perform single-fidelity, using EI
train_x_ei_list = []
train_obj_ei_list = []
cumulative_cost_list = []

for i_trial in range(10):

    cumulative_cost = 0.0
    init_x, init_obj = generate_initial_data(n=10)
    train_x_ei, train_obj_ei = init_x, init_obj

    while cumulative_cost < 150:
        mll, model = initialize_model(train_x_ei, train_obj_ei)
        fit_gpytorch_mll(mll)
        ei_acqf = get_ei(model, best_f=train_obj_ei.max())
        new_x, new_obj, cost = optimize_ei_and_get_observation(ei_acqf)
        train_x_ei = torch.cat([train_x_ei, new_x])
        train_obj_ei = torch.cat([train_obj_ei, new_obj])
        cumulative_cost += cost
    
    train_x_ei_list.append(train_x_ei)
    train_obj_ei_list.append(train_obj_ei)
    cumulative_cost_list.append(cumulative_cost)

# Calculate best curves and plot
from scipy.interpolate import interp1d

fixed_cost = 5.0

# Calculate cumulative best vs cost for EI, which has a constant cost
ei_cost_single = fixed_cost + 1
cost_ei = (np.arange(len(train_obj_ei_mean))+1) * ei_cost_single
# print(torch.cat(train_obj_ei_list, dim=1).shape)
# print(torch.cummax(torch.cat(train_obj_ei_list, dim=1), dim=0)[0].shape)
train_obj_ei_stddev, train_obj_ei_mean = torch.std_mean(torch.cummax(torch.cat(train_obj_ei_list, dim=1), dim=0)[0], dim=1)

# Calculate cumulative best vs cost for MF-KG, using recommendations
# plt.figure()
max_cost = np.max([x[-1] for x in cost_rec_list])
cost_plot = np.arange(0, max_cost, 0.25)
train_rec_plot_list = []
for train_x, train_obj, train_rec, cost_rec in \
    zip(train_x_list, train_obj_list, train_rec_list, cost_rec_list):

    # cost_plot 
    train_cost = train_x[:, -1] + fixed_cost
    cost_rec = np.concatenate((np.cumsum(train_cost[:10]), cost_rec))
    train_rec = np.concatenate((np.maximum.accumulate(train_obj[:10, 0]), np.maximum.accumulate(train_rec)))
    # We have to interpolate in x because they have different x
    train_rec_plot = interp1d(cost_rec, train_rec, kind='previous', bounds_error=False, 
        assume_sorted=True, fill_value=(np.nan, train_rec[-1]))(cost_plot)
    train_rec_plot_list.append(train_rec_plot)

    # plt.plot(cost_rec, train_rec)
    
# plt.figure()
# for train_rec_plot in train_rec_plot_list:
#     plt.plot(cost_plot, train_rec_plot)

# print(np.shape(train_rec_plot_list))
# # > (10, 878)

train_rec_plot_mean = np.nanmean(train_rec_plot_list, axis=0)
train_rec_plot_std = np.nanstd(train_rec_plot_list, axis=0)

plt.figure()
plt.plot(cost_ei, train_obj_ei_mean, label='SF-EI')
plt.fill_between(cost_ei, train_obj_ei_mean-train_obj_ei_stddev, train_obj_ei_mean+train_obj_ei_stddev, alpha=0.2)
plt.plot(cost_plot, train_rec_plot_mean, label='MF-KG')
plt.fill_between(cost_plot, train_rec_plot_mean-train_rec_plot_std, train_rec_plot_mean+train_rec_plot_std, alpha=0.2)
plt.legend()
plt.xlabel("Cost")
plt.ylabel("Best objective")

[Balandat]

Hmm so looking at that tutorial, it seems that the cost is 5.0 + s, where s \in {0.5, 0.75, 1.0}. This means that the cost difference between low and high fidelity observations is tiny and only ranges between 5.5 and 6.0. I haven't looked into this in more detail, but my hunch is that this is simply not a particularly meaningful problem for illustrating the benefits of multi-fidelity optimization (which begs the question why we have this particular setup as the example - I'll look into that). I'd recommend reducing the fixed cost to something much less (e.g. 0.25 or something like that) and trying this again (for a smaller total cost budget) and see how that changes the results - I'd be quite interested in seeing that.

[Balandat]

cc @SebastianAment, @ltiao, @dme65