Showcase: Validation

[alex4200]

Showcase Usecase

Aspect	Detail
Summary	Validation Use Case
Common document	https://docs.google.com/document/d/1sIOLMhK-jYnSb4x-xDEewmNqQXd4sztfw8nrgYsbZYs/edit#
Usecase Group
Expert	@apdavison
Scientific User
Deadline	March 2020
HPC Requirements
Dependencies
Nominal runtime

Description

This use case is supposed to be a showcase use case to be shown to the reviewers at the end of SGA2 in May/June 2020. This section should contain a detailed description of what exactly to show to the reviewers (like a script):

Acceptance Criteria

Define here the acceptance tests to evaluate the use case’s compliance with the requirements as defined above. Also possible end users for testing can be included here.

Extra Requirements

System

Does this use case require any special system requirements (any special software package)? Does this use case require HPC access? If so, which ones? List detailed system requirements here.

Performance

What are additional performance requirements for this use case? What time should it take to run from start to end?

[apdavison]

This should be for the M24 milestone, shouldn't it?

[alex4200]

@apdavison Yes you are correct - but to have the three Showcase items more visible I have them also in the September milestone (and they do not count in the weekly statistics).

[alex4200]

@appukuttan-shailesh @apdavison Can you please add a somewhat detailed description/script to this issue of what you want to show to the reviewers? This information is still missing.

[appukuttan-shailesh]

Description

We intend to demonstrate the HippoUnit validation use case. HippoUnit is a test suite developed for the validation of electrophysiological responses of hippocampal single cell models. The tests simulate CA1 pyramidal cell models that mimic the electrophysiological protocol corresponding to that originally employed for obtaining the target experimental data. The behavior of the model is then evaluate quantitatively and compared to the target data using feature-based error scores. It currently comprises the following tests:

• Somatic Features Test
  Evaluates the model for various electrophys Feature Extraction Library (eFEL) features under somatic current injection of varying amplitudes.
  Experimental data: Migliore et al. 2018

• Depolarization Block Test
  Aims to determine whether the model enters depolarization block under prolonged, high intensity somatic current stimulus.
  Experimental data: Bianchi et al. 2012

• Backpropagating Action Potential (AP) Test
  Evaluates the mode and efficacy of back-propagating action potentials on the apical trunk at different distances from the soma.
  Experimental data: Golding et al. 2001

• Post-Synaptic Potential (PSP) Attenuation Test
  Evaluates how much the post synaptic potential (using EPSC stimulus) attenuates from the dendrite (different distances) to the soma.
  Experimental data: Magee & Cook 2000

• Oblique Integration Test
  Tests signal integration in radial oblique dendrites for increasing number of synchronous and asynchronous inputs.
  Experimental data: Losonczy & Magee 2006

Acceptance Criteria

The Jupyter notebook should run to completion for the selected validations (see below: Steps to follow for demonstration). A scoring matrix and a validation report will be generated in the final few cells, which gives access to all available info on the validation results, and the underlying models and tests.

Extra Requirements

System

The following validation usecase specific Python packages are required for demonstration:

• sciunit >=0.2.1
• hippounit >= 1.3.3
• hbp_validation_framework >= 0.5.17

Performance

The first run of each validation (i.e. same test-model combination) can take on average 15 minutes for completion. Subsequent runs of the same validation, on the same machine with same user credentials, can be run with cached (default mode) data and thus would display the results faster. Therefore, for live demonstration purposes it would be advisable to run the desired the validation(s) beforehand, so as to allow live demo of the UseCase from start to end. Even with the usage of cached results, the validation would require around 5 minutes when all the tests are being run as only the raw data is cached, and all graphs and figures are re-created. Alternatively, we can launch the validations while live, and then show a previously completed validation result.
(@sasaray would be the expert for help with caching feature).

Note

Improvements are being made on several components of the validation framework:

• The model catalog and validation apps are being renovated for a much better user experience; should be ready by March. Current version is completely functional. (expert: @apdavison )
• Other minor updates being made to the Python packages to ensure Py3 compatibility and better features; should be ready by end of December.

Steps to follow for demonstration:

Following are the input steps:

• Step 3) Enter the # of required model: 1
  (Name: CA1_pyr_cACpyr_oh140807_A0_idG_20190328143451
  UUID: 055557a5-1bc6-4f7e-9816-981813596f9c)

• Step 4) Please choose from the following available tests for your cell type: all

• Step 5.2) Do you wish to generate a PDF report of the executed tests? Enter: y/n: Y

[alex4200]

Details see now #514