With repeated experiments, the TinyDB observer only logs the last one

[michalgregor]

I was trying to automatically run an experiment several times and log the results using the tinydb observer. The code of the experiment follows here (test.py):

import numpy as np
from sacred import Experiment
from sacred.observers import TinyDbObserver

ex = Experiment('normal_experiment')
ex.observers.append(TinyDbObserver.create('sacred_results'))

@ex.config
def cfg():
  mu = 0.0
  std = 1.0

@ex.automain
def run(mu, std):
    return np.random.normal(loc=mu, scale=std)

I run it 3 times with slightly different parameters like so:

from test import ex

ex.run()
ex.run(config_updates={'mu': 0.5})
ex.run(config_updates={'mu': 15})

When I inspect the database afterwards, there is only 1 entry corresponding to the last run.

I have also found that when I recreate the observer every time like so:

from test import ex
from sacred.observers import TinyDbObserver

ex.run()
del ex.observers[0]
ex.observers.append(TinyDbObserver.create('sacred_results'))

ex.run(config_updates={'mu': 0.5})
del ex.observers[0]
ex.observers.append(TinyDbObserver.create('sacred_results'))

ex.run(config_updates={'mu': 15})

all 3 entries are logged correctly.

I am not sure whether this is a bug, or whether I am perhaps not using the API as intended.

[thequilo]

I don't know whether this is a bug or not, but you can automate your workaround by setting the observer in a config scope:

import numpy as np
from sacred import Experiment
from sacred.observers import TinyDbObserver

ex = Experiment('normal_experiment')
ex.observers.append(None)

@ex.config()
def observer_config():
    ex.observers[0] = TinyDbObserver.create('sacred_results')

@ex.config
def cfg():
  mu = 0.0
  std = 1.0

@ex.automain
def run(mu, std):
    return np.random.normal(loc=mu, scale=std)

It could work to just reset the db_run_id for the observer (ex.observers[0].db_run_id = None). Maybe it is a bug that this id is not reset in started_event.

[Qwlouse]

This is definitely a bug, thanks for pointing this out!

[michalgregor]

Hi, I have tried adding self.db_run_id = None into started_event as suggested and this does seem to help.

I do not know the framework well enough to tell whether this is a good solution, but if so, maybe I could make a pull request to get it merged?