Best validation performance in multi-round training

[ali-akhavan89]

Hello,

I want to double-check my understanding of the best validation loss.

1. When doing multi-round SNPE training, the first validation loss is not comparable with the ones from the other rounds. Is my understanding correct?
2. Also, is there any meaningful information about the best validation loss for SNPE-C with atomic loss? For example, is 'Best validation performance: 0.02' in round 3 better than 'Best validation performance: 0.3' in round 11?
3. Finally, should I even compare the validation losses across training rounds and look for any patterns?

Update: I just learned the application of 'force_first_round_loss=True' and could see its impact on loss value (so I got my answer to my first question). But I would appreciate any feedback or comment anyone has. Thanks a lot!

Thank you,
Ali

[janfb]

Hi @ali-akhavan89

thanks for the questions - very good ones!

1. Indeed, as the loss function itself is different in the first round and in the following rounds, it is not comparable.
2. Also here it is difficult to compare the validation performance between the rounds. First, the validation set is slightly different (each round new data sampled from the current proposal is added). Second, the proposals change with every round especially in the first rounds when we are still "zooming in" to the region conditioned on x_o.
3. Given the above answers probably not. The validation performance is still useful though, e.g., for the early stopping criterion within a round and for comparing different neural architectures across entire runs (because for SNPE the validation loss is the negative log prob under the current posterior estimate).

Does this help?

Best,
Jan

[ali-akhavan89]

Thank you, Jan. These are indeed very helpful.