How to Save single Optimization Experiment

[StanleyYoo]

Hi,
I have followed Tutorial and tried to save the experiment by JSON but following error occurs: ax.exceptions.storage.JSONEncodeError: Object <main.MyRunner object at 0x0000016ED79986A0> passed to object_to_json (of type <class 'main.MyRunner'>, module: main) is not registered with a corresponding encoder in ENCODER_REGISTRY.

I have updated MyRunner class to save data as a record:

class MyRunner(Runner):
    def run(self, trial):
        # trial_metadata = {"name": str(trial.index)}
        # name_to_params = {arm.: arm.params for arm in trial.a}
        for arm_name, arm in trial.arms_by_name.items():
            records = []
            params = arm.parameters
            records.append(
                {
                    "arm_name": arm_name,
                    "x0": params["x0"],
                    "x1": params["x1"],
                    "x2": params["x2"],
                    "x3": params["x3"],
                    "x4": params["x4"],
                    "x5": params["x5"],
                    "trial_index": trial.index,
                }
            )
            df = pd.DataFrame.from_records(records)
        return df

Could you please let me know what causes an error?

[bernardbeckerman]

@StanleyYoo thanks for raising this here!

It seems like the error you received is indicating that the MyRunner class you're trying to save (i.e., the one attached to the experiment) is not the same as the MyRunner class in the registry (i.e., the MyRunner present in your notebook at the time of saving). I believe if you replace the previous MyRunner with your new MyRunner during runner definition here, and then re-run the notebook from the beginning, things should work.

As a quick follow-up, it seems like your runner may be duplicating some data that is already available elsewhere on the trial, and I believe some added context on what you're trying to accomplish may help me guide you to a better solution. For instance, perhaps exp_to_df could be helpful here, e.g., if you try

from ax.service.utils.report_utils import exp_to_df
exp_to_df(exp)

you may get the dataframe you're trying to create, but without having to alter the runner and pass around extraneous data. Let me know if that helps!

[StanleyYoo]

Hi @bernardbeckerman ! Thanks for your comments indeed.
Your suggestion resolved my concern of saving the outcomes more efficient way. Still my initial intention to use save_experiment is to restore the experiment after some numbers of GPEI and re-train with further trial once necessity arises. Still the same error published with following code even though MyRunner class has not been modified. Do you have any idea?

`class MyRunner(Runner):
def run(self, trial):
trial_metadata = {"name": str(trial.index)}
return trial_metadata

exp = Experiment(
name="test_hartmann",
search_space=hartmann_search_space,
optimization_config=optimization_config,
runner=MyRunner(),
)

NUM_SOBOL_TRIALS = 5
NUM_BOTORCH_TRIALS = 15

print(f"Running Sobol initialization trials...")
sobol = Models.SOBOL(search_space=exp.search_space)

for i in range(NUM_SOBOL_TRIALS):
# Produce a GeneratorRun from the model, which contains proposed arm(s) and other metadata
generator_run = sobol.gen(n=1)
# Add generator run to a trial to make it part of the experiment and evaluate arm(s) in it
trial = exp.new_trial(generator_run=generator_run)
# Start trial run to evaluate arm(s) in the trial
trial.run()
# Mark trial as completed to record when a trial run is completed
# and enable fetching of data for metrics on the experiment
# (by default, trials must be completed before metrics can fetch their data,
# unless a metric is explicitly configured otherwise)
trial.mark_completed()

for i in range(NUM_BOTORCH_TRIALS):
print(
f"Running GP+EI optimization trial {i + NUM_SOBOL_TRIALS + 1}/{NUM_SOBOL_TRIALS + NUM_BOTORCH_TRIALS}..."
)
# Reinitialize GP+EI model at each step with updated data.
gpei = Models.BOTORCH(experiment=exp, data=exp.fetch_data())
generator_run = gpei.gen(n=1)
trial = exp.new_trial(generator_run=generator_run)
trial.run()
trial.mark_completed()
save_experiment(exp, filepath)

print("Done!")`

Many thanks in advance.

[bernardbeckerman]

It seems like you're not including the encoder per section 9 of the tutorial. If instead of

save_experiment(exp, filepath)

I use

save_experiment(exp, filepath, encoder_registry=bundle.encoder_registry)

this succeeds. Closing this out but please feel free to open a new issue if you need more assistance : )