Tested DLC multicam pipeline

.github/workflows/codespell.yml → .github/workflows/lint.yml

@@ -1,13 +1,14 @@
----
-name: Codespell
+name: Lint
 
-on:
-  push:
-    branches: [master]
-  pull_request:
-    branches: [master]
+on: [push, pull_request]
 
 jobs:
+  black:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - uses: psf/black@stable
+
   codespell:
     name: Check for spelling errors
     runs-on: ubuntu-latest

.github/workflows/publish-docs.yml

@@ -40,7 +40,9 @@ jobs:
 
       - name: Deploy
         run: |
-          echo "OWNER: ${REPO_OWNER}"
+          FULL_VERSION=${{ github.ref }}
+          export MAJOR_VERSION=${FULL_VERSION:0:3}
+          echo "OWNER: ${REPO_OWNER}. BUILD: ${MAJOR_VERSION}"
           bash ./docs/build-docs.sh push $REPO_OWNER
         env:
           USERNAME: github-actions[bot]

.github/workflows/test-conda.yml

@@ -2,6 +2,8 @@ name: Test conda env and run tests
 
 on:
   push:
+    branches: 
+      - '!test_branch'
   schedule:  # once a day at midnight UTC
     - cron: '0 0 * * *'
 

.github/workflows/test-package-build.yml

@@ -5,6 +5,7 @@ on:
     branches:
       - master
       - maint/*
+      - '!test_branch'
     tags:
       - "*"
   pull_request:

CHANGELOG.md

@@ -1,30 +1,47 @@
 # Change Log
 
-## [0.4.4] (Unreleased)
+## Unreleased
 
 ### Infrastructure
 
-- Additional documentation. #690
-- Clean up following pre-commit checks. #688
-- Add Mixin class to centralize `fetch_nwb` functionality. #692, #734
-- Refactor restriction use in `delete_downstream_merge` #703
-- Add `cautious_delete` to Mixin class
-    - Initial implementation. #711, #762
-    - More robust caching of join to downstream tables. #806
-    - Overwrite datajoint `delete` method to use `cautious_delete`. #806
-    - Reverse join order for session summary. #821
-    - Add temporary logging of use to `common_usage`. #811, #821
-- Add `deprecation_factory` to facilitate table migration. #717
-- Add Spyglass logger. #730
-- IntervalList: Add secondary key `pipeline` #742
-- Increase pytest coverage for `common`, `lfp`, and `utils`. #743
-- Update docs to reflect new notebooks. #776
-- Add overview of Spyglass to docs. #779
-- Update linting for Black 24. #808
-- Steamline dependency management. #822
+- Add user roles to `database_settings.py`. #832
+
+## [0.5.0] (February 9, 2024)
+
+### Infrastructure
+
+- Docs:
+    - Additional documentation. #690
+    - Add overview of Spyglass to docs. #779
+    - Update docs to reflect new notebooks. #776
+- Mixin:
+    - Add Mixin class to centralize `fetch_nwb` functionality. #692, #734
+    - Refactor restriction use in `delete_downstream_merge` #703
+    - Add `cautious_delete` to Mixin class
+        - Initial implementation. #711, #762
+        - More robust caching of join to downstream tables. #806
+        - Overwrite datajoint `delete` method to use `cautious_delete`. #806
+        - Reverse join order for session summary. #821
+        - Add temporary logging of use to `common_usage`. #811, #821
+- Merge Tables:
+    - UUIDs: Revise Merge table uuid generation to include source. #824
+    - UUIDs: Remove mutual exclusivity logic due to new UUID generation. #824
+    - Add method for `merge_populate`. #824
+- Linting:
+    - Clean up following pre-commit checks. #688
+    - Update linting for Black 24. #808
+- Misc:
+    - Add `deprecation_factory` to facilitate table migration. #717
+    - Add Spyglass logger. #730
+    - Increase pytest coverage for `common`, `lfp`, and `utils`. #743
+    - Steamline dependency management. #822
 
 ### Pipelines
 
+- Common:
+    - `IntervalList`: Add secondary key `pipeline` #742
+    - Add `common_usage` table. #811, #821, #824
+    - Add catch errors during `populate_all_common`. #824
 - Spike sorting:
     - Add SpikeSorting V1 pipeline. #651
     - Move modules into spikesorting.v0 #807
@@ -159,4 +176,4 @@
 [0.4.1]: https://github.com/LorenFrankLab/spyglass/releases/tag/0.4.1
 [0.4.2]: https://github.com/LorenFrankLab/spyglass/releases/tag/0.4.2
 [0.4.3]: https://github.com/LorenFrankLab/spyglass/releases/tag/0.4.3
-[0.4.4]: https://github.com/LorenFrankLab/spyglass/releases/tag/0.4.4
+[0.5.0]: https://github.com/LorenFrankLab/spyglass/releases/tag/0.5.0

CITATION.cff

@@ -146,5 +146,5 @@ keywords:
   - spike sorting
   - kachery
 license: MIT
-version: 0.4.3
-date-released: '2023-11-07'
+version: 0.5.0
+date-released: '2024-02-09'

README.md

@@ -5,6 +5,8 @@
 
 ![Spyglass Figure](docs/src/images/fig1.png)
 
+[Demo](https://spyglass.hhmi.2i2c.cloud/hub/user-redirect/git-pull?repo=https%3A%2F%2Fgithub.com%2FLorenFrankLab%2Fspyglass-demo&urlpath=lab%2Ftree%2Fspyglass-demo%2Fnotebooks%2F01_Insert_Data.ipynb&branch=main) | [Installation](https://lorenfranklab.github.io/spyglass/latest/installation/) | [Docs](https://lorenfranklab.github.io/spyglass/) | [Tutorials](https://github.com/LorenFrankLab/spyglass/tree/master/notebooks) | [Citation](#citation)
+
 `spyglass` is a data analysis framework that facilitates the storage, analysis,
 visualization, and sharing of neuroscience data to support reproducible
 research. It is designed to be interoperable with the NWB format and integrates
@@ -81,11 +83,7 @@ License and Copyright notice can be found at
 
 ## Citation
 
-> Lee, K.H.\*, Denovellis, E.L.\*, Ly, R., Magland, J., Soules, J., Gramling, D.P.,
- Guidera, J.A., Nevers, R., Adenekan, P., Bray, S., et al. (2024). Spyglass:
-a data analysis framework for reproducible and shareable neuroscience research.
-   bioRxiv.
-   [10.1101/2024.01.25.577295](https://doi.org/10.1101/2024.01.25.577295 ).
+> Lee, K.H.\*, Denovellis, E.L.\*, Ly, R., Magland, J., Soules, J., Comrie, A.E., Gramling, D.P., Guidera, J.A., Nevers, R., Adenekan, P., Brozdowski, C., Bray, S., Monroe, E., Bak, J.H., Coulter, M.E., Sun, X., Tritt, A., Rübel, O., Nguyen, T., Yatsenko, D., Chu, J., Kemere, C., Garcia, S., Buccino, A., Frank, L.M., 2024. Spyglass: a data analysis framework for reproducible and shareable neuroscience research. bioRxiv. [10.1101/2024.01.25.577295](https://doi.org/10.1101/2024.01.25.577295 ).
 
 *\* Equal contribution*
 

config/add_dj_collaborator.py

@@ -9,4 +9,4 @@
         "This script is deprecated. "
         + "Use spyglass.utils.database_settings.DatabaseSettings instead."
     )
-    DatabaseSettings(user_name=sys.argv[1]).add_collab_user()
+    DatabaseSettings(user_name=sys.argv[1]).add_collab()

config/add_dj_guest.py

@@ -9,4 +9,4 @@
         "This script is deprecated. "
         + "Use spyglass.utils.database_settings.DatabaseSettings instead."
     )
-    DatabaseSettings(user_name=sys.argv[1]).add_dj_guest()
+    DatabaseSettings(user_name=sys.argv[1]).add_guest()

docs/build-docs.sh

@@ -1,8 +1,7 @@
 #!/bin/bash
 # Run this script from repo root to serve site: > bash ./docs/build-docs.sh serve
 # Then, navigate to localhost:8000/ to inspect site, then ctrl+c to exit
-# For auto-reload during dev, use `mkdocs serve -f ./docs/mkdosc.yaml`
-
+# For auto-reload during dev, use `mkdocs serve -f ./docs/mkdocs.yaml`
 
 # Copy top-level repo files for docs display
 cp ./CHANGELOG.md ./docs/src/
@@ -14,26 +13,29 @@ mv ./docs/src/notebooks/README.md ./docs/src/notebooks/index.md
 cp -r ./notebook-images ./docs/src/notebooks/
 cp -r ./notebook-images ./docs/src/
 
-# Get major version
-version_line=$(grep "__version__ =" ./src/spyglass/_version.py)
-version_string=$(echo "$version_line" | awk -F"[\"']" '{print $2}')
-export MAJOR_VERSION="${version_string:0:3}"
-echo "$MAJOR_VERSION"
+if [ -z "$MAJOR_VERSION" ]; then # Get version from file
+  version_line=$(grep "__version__ =" ./src/spyglass/_version.py)
+  version_string=$(echo "$version_line" | awk -F"[\"']" '{print $2}')
+  export MAJOR_VERSION="${version_string:0:3}"
+fi
+echo "$MAJOR_VERSION" # May be available as env var
 
 # Get ahead of errors
 export JUPYTER_PLATFORM_DIRS=1
-# jupyter notebook --generate-config
+jupyter notebook --generate-config -y &> /dev/null
+jupyter trust ./docs/src/notebooks/*.ipynb &> /dev/null
 
 # Generate site docs
-mike deploy "$MAJOR_VERSION" --config ./docs/mkdocs.yml -b documentation
+mike deploy "$MAJOR_VERSION" --config ./docs/mkdocs.yml -b documentation \
+  2>&1 | grep -v 'kernel_spec' # Suppress kernel_spec errors
 
 # Label this version as latest, set as default
-mike alias "$MAJOR_VERSION" latest --config ./docs/mkdocs.yml -b documentation
-mike set-default latest --config ./docs/mkdocs.yml -b documentation
+mike alias "$MAJOR_VERSION" latest -u --config ./docs/mkdocs.yml -b documentation
+# mike set-default latest --config ./docs/mkdocs.yml -b documentation
 
 # # Serve site to localhost
 if [ "$1" == "serve" ]; then # If first arg is serve, serve docs
-  mike serve --config ./docs/mkdocs.yml -b documentation
+  mike serve --config ./docs/mkdocs.yml -b documentation | grep -v 'kernel_spec'
 elif [ "$1" == "push" ]; then # if first arg is push
     if [ -z "$2" ]; then # When no second arg, use local git user
         git_user=$(git config user.name)

docs/mkdocs.yml

@@ -43,13 +43,6 @@ theme:
 nav:
   - Home: index.md
   - Installation: installation.md
-  - Miscellaneous:
-      - Overview: misc/index.md
-      - FigURL: misc/figurl_views.md
-      - Session Groups: misc/session_groups.md
-      - Insert Data: misc/insert_data.md
-      - Merge Tables: misc/merge_tables.md
-      - Database Management: misc/database_management.md
   - Tutorials:
       - Overview: notebooks/index.md
       - Intro:
@@ -61,7 +54,7 @@ nav:
       - Spikes:
           - Spike Sorting V0: notebooks/10_Spike_SortingV0.ipynb
           - Spike Sorting V1: notebooks/10_Spike_SortingV1.ipynb
-          - Curation: notebooks/11_Curation.ipynb
+          - Curation: notebooks/11_CurationV0.ipynb
       - Position:
           - Position Trodes: notebooks/20_Position_Trodes.ipynb
           - DLC Models: notebooks/21_DLC.ipynb
@@ -72,9 +65,17 @@ nav:
           - Theta: notebooks/31_Theta.ipynb
           - Ripple Detection: notebooks/32_Ripple_Detection.ipynb
       - Decoding:
-          - Extract Clusterless: notebooks/41_Extracting_Clusterless_Waveform_Features.ipynb
-          - Decoding Clusterless: notebooks/42_Decoding_Clusterless.ipynb
-          - Decoding Sorted Spikes: notebooks/43_Decoding_SortedSpikes.ipynb
+          - Extracting Waveforms: notebooks/40_Extracting_Clusterless_Waveform_Features.ipynb
+          - Decoding Clusterless: notebooks/41_Decoding_Clusterless.ipynb
+          - Decoding Sorted Spikes: notebooks/42_Decoding_SortedSpikes.ipynb
+      - MUA Detection: notebooks/50_MUA_Detection.ipynb
+  - Miscellaneous:
+      - Overview: misc/index.md
+      - FigURL: misc/figurl_views.md
+      - Session Groups: misc/session_groups.md
+      - Insert Data: misc/insert_data.md
+      - Merge Tables: misc/merge_tables.md
+      - Database Management: misc/database_management.md
   - API Reference: api/ # defer to gen-files + literate-nav
   - How to Contribute: contribute.md
   - Change Log: CHANGELOG.md

docs/src/index.md

@@ -68,12 +68,10 @@ contains license information.
 
 ## Citing Spyglass
 
-Kyu Hyun Lee, Eric Denovellis, Ryan Ly, Jeremy Magland, Jeff Soules, Alison
-Comrie, Jennifer Guidera, Rhino Nevers, Daniel Gramling, Philip Adenekan, Ji
-Hyun Bak, Emily Monroe, Andrew Tritt, Oliver Rübel, Thinh Nguyen, Dimitri
-Yatsenko, Joshua Chu, Caleb Kemere, Samuel Garcia, Alessio Buccino, Emily Aery
-Jones, Lisa Giocomo, and Loren Frank. 'Spyglass: A Data Analysis Framework for
-Reproducible and Shareable Neuroscience Research.' (2022) Society for
-Neuroscience, San Diego, CA.
+> Lee, K.H.\*, Denovellis, E.L.\*, Ly, R., Magland, J., Soules, J., Comrie, A.E., Gramling, D.P., Guidera, J.A., Nevers, R., Adenekan, P., Brozdowski, C., Bray, S., Monroe, E., Bak, J.H., Coulter, M.E., Sun, X., Tritt, A., Rübel, O., Nguyen, T., Yatsenko, D., Chu, J., Kemere, C., Garcia, S., Buccino, A., Frank, L.M., 2024. Spyglass: a data analysis framework for reproducible and shareable neuroscience research. bioRxiv. [10.1101/2024.01.25.577295](https://doi.org/10.1101/2024.01.25.577295 ).
+
+*\* Equal contribution*
+
+See paper related code [here](https://github.com/LorenFrankLab/spyglass-paper).
 
 <!-- TODO: Convert ccf file and insert here  -->

docs/src/misc/database_management.md

@@ -36,19 +36,28 @@ schema/database prefix.
 - `ALL` privileges allow users to create, alter, or drop tables and schemas in
     addition to operations above.
 
-In practice, DataJoint only permits alerations of secondary keys on existing
+In practice, DataJoint only permits alterations of secondary keys on existing
 tables, and more derstructive operations would require using DataJoint to
 execeute MySQL commands.
 
 Shared schema prefixes are those defined in the Spyglass package (e.g.,
 `common`, `lfp`, etc.). A 'user schema' is any schema with the username as
 prefix. User types differ in the privileges they are granted on these prifixes.
+Declaring a table with the SpyglassMixin on a schema other than a shared module
+or the user's own prefix will raise a warning.
 
-### Users types
+### Users roles
+
+When a database is first initialized, the team should run `add_roles` to create
+the following roles:
 
-- `collab_user`: `ALL` on user schema, `SELECT` on all other schemas.
 - `dj_guest`: `SELECT` on all schemas.
+- `dj_collab`: `ALL` on user schema, `SELECT` on all other schemas.
 - `dj_user`: `ALL` on shared and user schema, `SELECT` on all other schemas.
+- `dj_admin`: `ALL` on all schemas.
+
+If new shared modules are introduced, the `add_module` method should be used to
+expand the privileges of the `dj_user` role.
 
 ### Setting Passwords
 
@@ -224,9 +233,19 @@ To remove orphaned files, we run the following commands in our cron jobs:
 ```python
 from spyglass.common import AnalysisNwbfile
 from spyglass.spikesorting import SpikeSorting
+from spyglass.common.common_nwbfile import schema as nwbfile_schema
+from spyglass.decoding.v1.sorted_spikes import schema as spikes_schema
+from spyglass.decoding.v1.clusterless import schema as clusterless_schema
 
 
 def main():
     AnalysisNwbfile().nightly_cleanup()
     SpikeSorting().nightly_cleanup()
+    nwbfile_schema.external['analysis'].delete(delete_external_files=True))
+    nwbfile_schema.external['raw'].delete(delete_external_files=True))
+    spikes_schema.external['analysis'].delete(delete_external_files=True))
+    clusterless_schema.external['analysis'].delete(delete_external_files=True))
 ```
+
+The `delete` calls above use DataJoint's `ExternalTable.delete` method, which
+will remove files from disk that are no longer referenced in the database.

environment_dlc.yml

@@ -22,7 +22,6 @@ dependencies:
   - libgcc   # dlc-only
   - matplotlib
   - non_local_detector
-  - numpy<1.24
   - pip>=20.2.*
   - position_tools
   - pybind11  # req by mountainsort4 -> isosplit5
@@ -47,4 +46,5 @@ dependencies:
       - pynwb>=2.2.0,<3
       - sortingview>=0.11
       - spikeinterface>=0.98.2,<0.99
+      - tensorflow<=2.12  # dlc-only
       - .[dlc]