Simulators test suite

[wvangeit]

As can be seen at:
#54
#15
reproducing sonata-specified simulations exactly is not a trivial task.

I propose that we make a set of tests of increasing complexity to test simulators that try to implement SONATA, so that we debug issues in a more systematic manner. The input data for each step is a SONATA circuit with stimulus config, the output is the expected data.

For detailed models I would propose the following (to be extended) procedure :
1-cell (compare voltage):

• 1-cell circuit, single soma, only cm / hh mechanism, 1 step current
• previous + active MOD file channels
• previous + 1 synapse on the soma, articifial activation of the synapse
• previous + synapses of several types on the soma
• all of the above with a more detailed morphology, and synapses located all over the morph

10 cell (compare voltage):

• small (e.g. 10 cell) cell circuit, detailed cells, step current injections, unconnected.
• previous, no step currents, but synapses get input 'as if' cells are connected, all types of synapses present.
• previous, connect all cells with real connections.

100 cell (compare raster):

• larger (e.g. 100 cell) cell circuit, all cell types present, step current injections, unconnected.
• previous, connected

We obviously need something similar for point neurons etc.

This is just an initial proposal, we should add tests as we go and get more experience with the things that can go wrong.

Once we have build such a test set, I think it will make debugging task of the simulator developers easier.

As an extra advantage it can also be used in the continuous integration of simulators later on.

I would say once we get 2 simulators to agree on the test set output, we can consider this task as done.

[pgleeson]

I agree completely with this!

Regarding the point neuron tests, the only available examples seem to be either NEURON only (using inbuilt IntFire1 etc) and NEST specific (using iaf_psc_alpha). Apart from the fact that the same model can't be run on both simulators, each of these comes with their own issues, e.g. default params in NEST, #45; lack of access to membrane potential in NEURON #55.

As mentioned previously, specifying the point neuron models in PyNN (with all parameter values explicitly listed) or NeuroML would address these problems.

[antonarkhipov]

@wvangeit , this is a great suggestion. Would you be willing to oversee and spearhead the implementation of these tests? That will be a very valuable contribution. I'm sure everyone here will be happy to help, but someone needs to drive it and make sure things are implemented in a timely fashion.

One thing to keep in mind, as we often discuss, is that we shouldn't expect an ideal match between multiple simulators. That is not expected even between some of the different versions of NEURON. But for sure it will be important to have these reference examples and to know what one should expect at least when using a particular simulator.

[wvangeit]

@antonarkhipov yep, sure I will start by adding some example circuits.
Next week I'm off though, so it will be something for after that.

[pgleeson]

I'm increasingly feeling that there's no point in having simulator specific examples in the simulator test suite, i.e. an I&F example that can only run on NEURON and a different impl of an I&F for NEST... An example with an I&F cell specified in PyNN that can be used for both would be ideal.

This addresses one potential criticism of Sonata in its currently state: there is no example network that can be run directly in more than one simulator (as opposed to changing to another format; correct me if I'm wrong...).