Merge pull request #50 from kabilar/main

Add NWB export pytests

Author: Thinh Nguyen

README.md

@@ -1,8 +1,8 @@
 # DataJoint Workflow - Array Electrophysiology
 
-Workflow for extracellular array electrophysiology data acquired with a polytrode probe (e.g.
-Neuropixels, Neuralynx) using the `SpikeGLX` or `OpenEphys` acquisition software and processed
-with MATLAB- or python-based `Kilosort` spike sorting software.
+Workflow for extracellular array electrophysiology data acquired with a polytrode probe 
+(e.g. [Neuropixels](https://www.neuropixels.org), Neuralynx) using the [SpikeGLX](https://github.com/billkarsh/SpikeGLX) or 
+[OpenEphys](https://open-ephys.org/gui) acquisition software and processed with [MATLAB-based Kilosort](https://github.com/MouseLand/Kilosort) or [python-based Kilosort](https://github.com/MouseLand/pykilosort) spike sorting software.
 
 A complete electrophysiology workflow can be built using the DataJoint Elements.
 + [element-lab](https://github.com/datajoint/element-lab)
@@ -18,34 +18,43 @@ convention, and directory lookup methods (see
 [workflow_array_ephys/paths.py](workflow_array_ephys/paths.py)).
 3. Ingestion of clustering results.
 
-## Workflow architecture
-
-The electrophysiology workflow presented here uses components from 4 DataJoint 
-Elements (`element-lab`, `element-animal`, `element-session`,
-`element-array-ephys`) assembled together to form a fully functional workflow.
-
-### element-lab
+See the [Element Array Electrophysiology documentation](https://elements.datajoint.org/description/array_ephys/) for the background information and development timeline.
 
-![element-lab](
-https://github.com/datajoint/element-lab/raw/main/images/lab_diagram.svg)
+For more information on the DataJoint Elements project, please visit https://elements.datajoint.org.  This work is supported by the National Institutes of Health.
 
-### element-animal
-
-![element-animal](
-https://github.com/datajoint/element-animal/raw/main/images/subject_diagram.svg)
+## Workflow architecture
 
-### assembled with element-array-ephys
+The electrophysiology workflow presented here uses components from 4 DataJoint 
+Elements ([element-lab](https://github.com/datajoint/element-lab), 
+[element-animal](https://github.com/datajoint/element-animal), 
+[element-session](https://github.com/datajoint/element-session),
+[element-array-ephys](https://github.com/datajoint/element-array-ephys)) 
+assembled together to form a fully functional workflow.
 
 ![element-array-ephys](images/attached_array_ephys_element.svg)
 
 ## Installation instructions
 
 + The installation instructions can be found at the 
-[datajoint-elements repository](https://github.com/datajoint/datajoint-elements/blob/main/gh-pages/docs/install.md).
+[DataJoint Elements documentation](https://elements.datajoint.org/usage/install/).
 
 ## Interacting with the DataJoint workflow
 
 + Please refer to the following workflow-specific 
  [Jupyter notebooks](/notebooks) for an in-depth explanation of how to run the 
  workflow ([03-process.ipynb](notebooks/03-process.ipynb)) and explore the data 
  ([05-explore.ipynb](notebooks/05-explore.ipynb)).
+
+## Citation
+
++ If your work uses DataJoint and DataJoint Elements, please cite the respective Research Resource Identifiers (RRIDs) and manuscripts.
+
++ DataJoint for Python or MATLAB
+    + Yatsenko D, Reimer J, Ecker AS, Walker EY, Sinz F, Berens P, Hoenselaar A, Cotton RJ, Siapas AS, Tolias AS. DataJoint: managing big scientific data using MATLAB or Python. bioRxiv. 2015 Jan 1:031658. doi: https://doi.org/10.1101/031658
+
+    + DataJoint ([RRID:SCR_014543](https://scicrunch.org/resolver/SCR_014543)) - DataJoint for `<Select Python or MATLAB>` (version `<Enter version number>`)
+
++ DataJoint Elements
+    + Yatsenko D, Nguyen T, Shen S, Gunalan K, Turner CA, Guzman R, Sasaki M, Sitonic D, Reimer J, Walker EY, Tolias AS. DataJoint Elements: Data Workflows for Neurophysiology. bioRxiv. 2021 Jan 1. doi: https://doi.org/10.1101/2021.03.30.437358
+
+    + DataJoint Elements ([RRID:SCR_021894](https://scicrunch.org/resolver/SCR_021894)) - Element Array Electrophysiology (version `<Enter version number>`)

docker/Dockerfile.dev

@@ -9,13 +9,15 @@ RUN mkdir /main/element-lab \
           /main/element-animal \
           /main/element-session \
           /main/element-array-ephys \
+          /main/element-interface \
           /main/workflow-array-ephys
 
 # Copy user's local fork of elements and workflow
 COPY --chown=anaconda:anaconda ./element-lab /main/element-lab
 COPY --chown=anaconda:anaconda ./element-animal /main/element-animal
 COPY --chown=anaconda:anaconda ./element-session /main/element-session
 COPY --chown=anaconda:anaconda ./element-array-ephys /main/element-array-ephys
+COPY --chown=anaconda:anaconda ./element-interface /main/element-interface
 COPY --chown=anaconda:anaconda ./workflow-array-ephys /main/workflow-array-ephys
 
 # Install packages
@@ -24,6 +26,7 @@ RUN pip install -e /main/element-animal
 RUN pip install -e /main/element-session
 RUN pip install -e /main/element-array-ephys
 RUN pip install -e /main/workflow-array-ephys
+RUN pip install -e /main/element-interface
 RUN pip install -r /main/workflow-array-ephys/requirements_test.txt
 
 WORKDIR /main/workflow-array-ephys

docker/docker-compose-dev.yaml

@@ -29,6 +29,7 @@ services:
       - ../../element-animal:/main/element-animal
       - ../../element-session:/main/element-session
       - ../../element-array-ephys:/main/element-array-ephys
+      - ../../element-interface:/main/element-interface
       - ..:/main/workflow-array-ephys
     depends_on:
       db:

tests/__init__.py

@@ -20,8 +20,8 @@
 _tear_down = True
 verbose = False
 
-test_user_data_dir = pathlib.Path('./tests/user_data')
-test_user_data_dir.mkdir(exist_ok=True)
+pathlib.Path('./tests/user_data').mkdir(exist_ok=True)
+pathlib.Path('./tests/user_data/lab').mkdir(exist_ok=True)
 
 sessions_dirs = ['subject1/session1',
                  'subject2/session1',
@@ -34,6 +34,17 @@
 # --------------------  HELPER CLASS --------------------
 
 
+def write_csv(content, path):
+    """
+    General function for writing strings to lines in CSV
+    :param path: pathlib PosixPath
+    :param content: list of strings, each as row of CSV
+    """
+    with open(path, 'w') as f:
+        for line in content:
+            f.write(line+'\n')
+
+
 class QuietStdOut:
     """If verbose set to false, used to quiet tear_down table.delete prints"""
     def __enter__(self):
@@ -121,6 +132,147 @@ def pipeline():
             pipeline.subject.Subject.delete()
 
 
+@pytest.fixture
+def lab_csv():
+    """ Create a 'labs.csv' file"""
+    lab_content = ["lab,lab_name,institution,address,"
+                   + "time_zone,location,location_description",
+                   "LabA,The Example Lab,Example Uni,"
+                   + "'221B Baker St,London NW1 6XE,UK',UTC+0,"
+                   + "Example Building,'2nd floor lab dedicated to all "
+                   + "fictional experiments.'",
+                   "LabB,The Other Lab,Other Uni,"
+                   + "'Oxford OX1 2JD, United Kingdom',UTC+0,"
+                   + "Other Building,'fictional campus dedicated to imaginary"
+                   + "experiments.'"]
+    lab_csv_path = pathlib.Path('./tests/user_data/lab/labs.csv')
+    write_csv(lab_content, lab_csv_path)
+
+    yield lab_content, lab_csv_path
+    lab_csv_path.unlink()
+
+
+@pytest.fixture
+def lab_project_csv():
+    """ Create a 'projects.csv' file"""
+    lab_project_content = ["project,project_description,repository_url,"
+                           + "repository_name,codeurl",
+                           "ProjA,Example project to populate element-lab,"
+                           + "https://github.com/datajoint/element-lab/,"
+                           + "element-lab,https://github.com/datajoint/element"
+                           + "-lab/tree/main/element_lab",
+                           "ProjB,Other example project to populate element-"
+                           + "lab,https://github.com/datajoint/element-session"
+                           + "/,element-session,https://github.com/datajoint/"
+                           + "element-session/tree/main/element_session"]
+    lab_project_csv_path = pathlib.Path('./tests/user_data/lab/projects.csv')
+    write_csv(lab_project_content, lab_project_csv_path)
+
+    yield lab_project_content, lab_project_csv_path
+    lab_project_csv_path.unlink()
+
+
+@pytest.fixture
+def lab_project_users_csv():
+    """ Create a 'project_users.csv' file"""
+    lab_project_user_content = ["user,project",
+                                "Sherlock,ProjA",
+                                "Sherlock,ProjB",
+                                "Watson,ProjB",
+                                "Dr. Candace Pert,ProjA",
+                                "User1,ProjA"]
+    lab_project_user_csv_path = pathlib.Path('./tests/user_data/lab/\
+                                              project_users.csv')
+    write_csv(lab_project_user_content, lab_project_user_csv_path)
+
+    yield lab_project_user_content, lab_project_user_csv_path
+    lab_project_user_csv_path.unlink()
+
+
+@pytest.fixture
+def lab_publications_csv():
+    """ Create a 'publications.csv' file"""
+    lab_publication_content = ["project,publication",
+                               "ProjA,arXiv:1807.11104",
+                               "ProjA,arXiv:1807.11104v1"]
+    lab_publication_csv_path = pathlib.Path('./tests/user_data/lab/\
+                                             publications.csv')
+    write_csv(lab_publication_content, lab_publication_csv_path)
+
+    yield lab_publication_content, lab_publication_csv_path
+    lab_publication_csv_path.unlink()
+
+
+@pytest.fixture
+def lab_keywords_csv():
+    """ Create a 'keywords.csv' file"""
+    lab_keyword_content = ["project,keyword",
+                           "ProjA,Study",
+                           "ProjA,Example",
+                           "ProjB,Alternate"]
+    lab_keyword_csv_path = pathlib.Path('./tests/user_data/lab/keywords.csv')
+    write_csv(lab_keyword_content, lab_keyword_csv_path)
+
+    yield lab_keyword_content, lab_keyword_csv_path
+    lab_keyword_csv_path.unlink()
+
+
+@pytest.fixture
+def lab_protocol_csv():
+    """ Create a 'protocols.csv' file"""
+    lab_protocol_content = ["protocol,protocol_type,protocol_description",
+                            "ProtA,IRB expedited review,Protocol for managing "
+                            + "data ingestion",
+                            "ProtB,Alternative Method,Limited protocol for "
+                            + "piloting only"]
+    lab_protocol_csv_path = pathlib.Path('./tests/user_data/lab/protocols.csv')
+    write_csv(lab_protocol_content, lab_protocol_csv_path)
+
+    yield lab_protocol_content, lab_protocol_csv_path
+    lab_protocol_csv_path.unlink()
+
+
+@pytest.fixture
+def lab_user_csv():
+    """ Create a 'users.csv' file"""
+    lab_user_content = ["lab,user,user_role,user_email,user_cellphone",
+                        "LabA,Sherlock,PI,Sherlock@BakerSt.com,"
+                        + "+44 20 7946 0344",
+                        "LabA,Watson,Dr,DrWatson@BakerSt.com,+44 73 8389 1763",
+                        "LabB,Dr. Candace Pert,PI,Pert@gmail.com,"
+                        + "+44 74 4046 5899",
+                        "LabA,User1,Lab Tech,fake@email.com,+44 1632 960103",
+                        "LabB,User2,Lab Tech,fake2@email.com,+44 1632 960102"]
+    lab_user_csv_path = pathlib.Path('./tests/user_data/lab/users.csv')
+    write_csv(lab_user_content, lab_user_csv_path)
+
+    yield lab_user_content, lab_user_csv_path
+    lab_user_csv_path.unlink()
+
+
+@pytest.fixture
+def ingest_lab(pipeline, lab_csv, lab_project_csv, lab_publications_csv,
+               lab_keywords_csv, lab_protocol_csv, lab_user_csv,
+               lab_project_users_csv):
+    """ From workflow_array_ephys ingest.py, import ingest_lab, run """
+    from workflow_array_ephys.ingest import ingest_lab
+    _, lab_csv_path = lab_csv
+    _, lab_project_csv_path = lab_project_csv
+    _, lab_publication_csv_path = lab_publications_csv
+    _, lab_keyword_csv_path = lab_keywords_csv
+    _, lab_protocol_csv_path = lab_protocol_csv
+    _, lab_user_csv_path = lab_user_csv
+    _, lab_project_user_csv_path = lab_project_users_csv
+    ingest_lab(lab_csv_path=lab_csv_path,
+               project_csv_path=lab_project_csv_path,
+               publication_csv_path=lab_publication_csv_path,
+               keyword_csv_path=lab_keyword_csv_path,
+               protocol_csv_path=lab_protocol_csv_path,
+               users_csv_path=lab_user_csv_path,
+               project_user_csv_path=lab_project_user_csv_path, verbose=verbose)
+    return
+
+
 @pytest.fixture
 def subjects_csv():
     """ Create a 'subjects.csv' file"""
@@ -160,11 +312,23 @@ def ingest_subjects(pipeline, subjects_csv):
 @pytest.fixture
 def sessions_csv(test_data):
     """ Create a 'sessions.csv' file"""
-    input_sessions = pd.DataFrame(columns=['subject', 'session_dir'])
+    input_sessions = pd.DataFrame(columns=['subject', 'session_dir', 'session_note', 
+                                           'user'])
     input_sessions.subject = ['subject1', 'subject2', 'subject2',
                               'subject3', 'subject4', 'subject5',
                               'subject6']
     input_sessions.session_dir = sessions_dirs
+    input_sessions.session_note = ['Data collection notes',
+                                   'Data collection notes',
+                                   'Interrupted session',
+                                   'Data collection notes',
+                                   'Successful data collection',
+                                   'Successful data collection',
+                                   'Ambient temp abnormally low']
+    input_sessions.user = ['User2', 'User2', 'User2',
+                           'User1', 'User2', 'User1',
+                           'User2']
+
     input_sessions = input_sessions.set_index('subject')
 
     sessions_csv_path = pathlib.Path('./tests/user_data/sessions.csv')
@@ -211,7 +375,7 @@ def ephys_insertionlocation(pipeline, ingest_sessions):
                                              depth=0,
                                              theta=0,
                                              phi=0,
-                                             beta=0))
+                                             beta=0), skip_duplicates=True)
     yield
 
     if _tear_down: