More notebook revisions (#616)

* Restructure for mkdocs * Minor docstring edits for mkdocs * Adjust installer docs * Permit json config * Update docstrings for mkdocs. Add images * Minor fixes. Rename publish action. Add changelog notes * hard wrap changes, markdownlint * blackify * Fix version cmd. Minor doc wording * mkdocstrings require empty inits to identify submodules * 🤦‍♂️ Publish docs CI/CD `main`->`master` * Edit `get_part`; Add `merge_fetch` * Spelling fixes * Refactor restrict_parts. Adjust mkdocs nav. * Docs adjust for Merge tables * Fix `merge_get_part` * Use hatch for docs version * Update changelog * Typo * Spellcheck config * WIP: dict/str ristrict consistency * Normalize restriction/classmeth. Add notes on why * Typos * Docs publish on tag * Edit changelog: Add links, patch version bump * 🧪 Test gh-actions debug * get-part multi-source flag * Add mutual exclusivity flag from pos branch * See details. Notebook work, config overhaul gitignore: add exclude example config mkdocs: new notebook names notebooks: complete revamp for minirec data and more links to docs init: add new load_config, isort imports common_lab: - adjust to accept names in Last, First format (nwb-compliant) - continue to use First Last name structure in database - yes? common_nwb: use new load_config, change `assert` to `raise` insert_sessions: permit paths, use file name, use raw dir storage_dirs: remove redundant funcs for base_dir settings: implement new base_dir system - allows base/raw/etc to be independent - defaults to dj.config, then env vars, then sets default rel paths * WIP: fix failing tests related to base_dir edits * underscore-prefix Merge. Linter fixes * See Details. Notebook overhaul - dj_config: accept base dir as arg, refactor for single responsibility - mkdocs, installation.md - condense installation information to single page - reference new notebook - remove local and production subpages due to redundancy - environment and env_position.yml: add install current dir to avoid additional step in installation process - notebooks: rewrite with Docker optional and minirec as demo data - common_lab: raise error for invalid name - common_position: get raw dir from settings, not hardcode - settings.py: Should this be a class with properties? - add options for kachery dirs set via same dj_config mechanism - add raw_dir helper function * remove note to self * WIP notebook edits * Revise 04_LFP nb * Reorder/revise notebooks; #609 * Notebook formatting * Remove old * jupytext backup note * Blackify py scrips. Continue config changes * WIP: notebooks, plus improved merge_delete_downstream * WIP: PositionSource add part table * Refactor trodes position #613 * WIP: Fix Trodes Video * WIP: Spellcheck. Remove debug params. Remove assigned lambda E713 * WIP: Pass tests. Remove codespell offending link * WIP: blackify * Selective fetch from cbroz/master * Fetch additional file from cbroz1/master to pass CI/CD * Add RawPos fetch method implementations. Object -> PosObject * Refactor PosIntervalMap helpers * Revert typo * Bugfixes for ripple * Blackify * WIP: minor edits * Add restriction to fetch1_dataframe * Update Trodes notebook, revise others * Set pos id default for migration. Rename Trodes params * Fix typos * Spelling; Jupytext sync; Blackify * Edit gitignore for new notebook numbering * Update changelog and notebooks * Refactor position helpers * Minor position_trodes fixes
LorenFrankLab · Oct 10, 2023 · 99d8f00 · 99d8f00
1 parent ce0bca4
commit 99d8f00
Show file tree

Hide file tree

Showing 53 changed files with 12,363 additions and 12,482 deletions.
diff --git a/.gitignore b/.gitignore
@@ -6,23 +6,21 @@ mysql_config
 
 # Notebooks
 *.ipynb
-
-!notebooks/00_intro.ipynb
-!notebooks/01_spikesorting.ipynb
-!notebooks/02_curation.ipynb
-!notebooks/03_lfp.ipynb
-!notebooks/04_Trodes_position.ipynb
-!notebooks/05_DLC_from_scratch.ipynb
-!notebooks/06_DLC_from_dir.ipynb
-!notebooks/4_position_info.ipynb
-!notebooks/07_linearization.ipynb
-!notebooks/08_Extract_Mark_indicators.ipynb
-!notebooks/09_Decoding_with_GPUs_on_the_GPU_cluster.ipynb
-!notebooks/10_1D_Clusterless_Decoding.ipynb
-!notebooks/11_2D_Clusterless_Decoding.ipynb
-!notebooks/12_Ripple_Detection.ipynb
-!notebooks/13_Theta_phase_and_power.ipynb
-
+!notebooks/00*.ipynb
+!notebooks/01*.ipynb
+!notebooks/02*.ipynb
+!notebooks/10*.ipynb
+!notebooks/11*.ipynb
+!notebooks/12*.ipynb
+!notebooks/14*.ipynb
+!notebooks/20*.ipynb
+!notebooks/21*.ipynb
+!notebooks/22*.ipynb
+!notebooks/23*.ipynb
+!notebooks/24*.ipynb
+!notebooks/30*.ipynb
+!notebooks/31*.ipynb
+!notebooks/33*.ipynb
 
 # Byte-compiled / optimized / DLL files
 __pycache__/
@@ -167,7 +165,7 @@ temp_nwb/*s
 *.json
 *.gz
 *.pdf
-dj_local_con*.json
+dj_local_conf*
 !dj_local_conf_example.json
 
 !/.vscode/extensions.json

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,5 +1,30 @@
 # Change Log
 
+## [0.4.2] (October 10, 2023)
+
+### Infrastructure / Support
+
+- Bumped Python version to 3.9. #583
+- Updated user management helper scripts for MySQL 8. #650
+- Centralized config/path handling to permit setting via datajoint config. #593
+- Fixed Merge Table deletes: error specificity and transaction context. #617
+
+### Pipelines
+
+- Common:
+  - Added support multiple cameras per epoch. #557
+  - Removed `common_backup` schema. #631
+  - Added support for multiple position objects per NWB in `common_behav` via
+    PositionSource.SpatialSeries and RawPosition.PosObject #628, #616.
+    _Note:_ Existing functions have been made compatible, but column labels for
+    `RawPosition.fetch1_dataframe` may change.
+- Spike sorting:
+  - Added pipeline populator. #637, #646, #647
+  - Fixed curation functionality for `nn_isolation`. #597, #598
+- Position: Added position interval/epoch mapping via PositionIntervalMap. #620,
+  #621, #627
+- LFP: Refactored pipeline. #594, #588, #605, #606, #607, #608, #615, #629
+
 ## [0.4.1] (June 30, 2023)
 
 - Add mkdocs automated deployment. #527, #537, #549, #551
@@ -62,6 +87,7 @@
 - Allow creation and linkage of device metadata from YAML #400
 - Move helper functions to utils directory #386
 
+[0.4.2]: https://github.com/LorenFrankLab/spyglass/releases/tag/0.4.2
 [0.4.1]: https://github.com/LorenFrankLab/spyglass/releases/tag/0.4.1
 [0.4.0]: https://github.com/LorenFrankLab/spyglass/releases/tag/0.4.0
 [0.3.4]: https://github.com/LorenFrankLab/spyglass/releases/tag/0.3.4

diff --git a/dj_local_conf_example.json b/dj_local_conf_example.json
@@ -31,6 +31,10 @@
     "database.prefix": "username_",
     "spyglass_dirs": {
       "base": "/your/base/path"
-    }
+    },
+    "kachery_dirs": {
+      "cloud": "/your/base/path/.kachery_cloud"
+    },
+    "kachery_zone": "franklab.default"
   }
 }
diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml
@@ -47,28 +47,30 @@ nav:
   - Home: index.md
   - Installation: installation.md
   - Miscellaneous:
-      - FigURL: misc/figurl_views.md
-      - Session Groups: misc/session_groups.md
-      - Insert Data: misc/insert_data.md
-      - Merge Tables: misc/merge_tables.md
+    - FigURL: misc/figurl_views.md
+    - Session Groups: misc/session_groups.md
+    - Insert Data: misc/insert_data.md
+    - Merge Tables: misc/merge_tables.md
   - Tutorials:
+    - General:
       - Setup: notebooks/00_Setup.ipynb
       - Insert Data: notebooks/01_Insert_Data.ipynb
-      - Spike Sorting: notebooks/02_Spike_Sorting.ipynb
-      - Curation: notebooks/03_Curation.ipynb
-      - LFP: notebooks/03_lfp.ipynb
-      - Position:
-          - Information: notebooks/4_position_info.ipynb
-          - Pipeline: notebooks/04_Trodes_position.ipynb
-          - From Scratch: notebooks/05_DLC_from_scratch.ipynb
-          - From Pre-Trained: notebooks/06_DLC_from_dir.ipynb
-      - Linearization Pipeline: notebooks/07_linearization.ipynb
-      - Mark Indicators: notebooks/08_Extract_Mark_indicators.ipynb
-      - GPU: notebooks/09_Decoding_with_GPUs_on_the_GPU_cluster.ipynb
-      - 1D Clusterless Decoding: notebooks/10_1D_Clusterless_Decoding.ipynb
-      - 2D Clusterless Decoding: notebooks/11_2D_Clusterless_Decoding.ipynb
-      - Ripple Detection: notebooks/12_Ripple_Detection.ipynb
-      - Spyglass Kachery Setup: notebooks/Spyglass_kachery_setup.ipynb
+      - Data Sync: notebooks/02_Data_Sync.ipynb
+    - Ephys: 
+      - Spike Sorting: notebooks/10_Spike_Sorting.ipynb
+      - Curation: notebooks/11_Curation.ipynb
+      - LFP: notebooks/12_LFP.ipynb
+      - Theta: notebooks/14_Theta.ipynb
+    - Position: 
+      - Position Trodes: notebooks/20_Position_Trodes.ipynb
+      - Position DLC 1: notebooks/21_Position_DLC_1.ipynb
+      - Position DLC 2: notebooks/22_Position_DLC_2.ipynb
+      - Linearization: notebooks/24_Linearization.ipynb
+    - Combined: 
+      - Ripple Detection: notebooks/30_Ripple_Detection.ipynb
+      - Extract Mark Indicators: notebooks/31_Extract_Mark_Indicators.ipynb
+      - Decoding with GPUs: notebooks/32_Decoding_with_GPUs.ipynb
+      - Decoding Clusterless: notebooks/33_Decoding_Clusterless.ipynb
   - API Reference: api/ # defer to gen-files + literate-nav
   - How to Contribute: contribute.md
   - Change Log: CHANGELOG.md

diff --git a/notebooks/00_Setup.ipynb b/notebooks/00_Setup.ipynb
@@ -231,19 +231,54 @@
    "source": [
     "### Loading the config\n",
     "\n",
-    "We can check that the paths are correctly set up by loading the config.\n"
+    "We can check that the paths are correctly set up by loading the config from \n",
+    "the main Spyglass directory.\n"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 1,
    "id": "912ac84b",
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'debug_mode': True,\n",
+       " 'prepopulate': True,\n",
+       " 'SPYGLASS_BASE_DIR': '/stelmo/nwb',\n",
+       " 'SPYGLASS_RAW_DIR': '/stelmo/nwb/raw',\n",
+       " 'SPYGLASS_ANALYSIS_DIR': '/stelmo/nwb/analysis',\n",
+       " 'SPYGLASS_RECORDING_DIR': '/stelmo/nwb/recording',\n",
+       " 'SPYGLASS_SORTING_DIR': '/stelmo/nwb/spikesorting',\n",
+       " 'SPYGLASS_WAVEFORMS_DIR': '/stelmo/nwb/waveforms',\n",
+       " 'SPYGLASS_TEMP_DIR': '/stelmo/nwb/tmp',\n",
+       " 'SPYGLASS_VIDEO_DIR': '/stelmo/nwb/video',\n",
+       " 'KACHERY_CLOUD_DIR': '/stelmo/nwb/kachery_storage',\n",
+       " 'KACHERY_STORAGE_DIR': '/stelmo/nwb/kachery_storage',\n",
+       " 'KACHERY_TEMP_DIR': '/stelmo/nwb/tmp',\n",
+       " 'KACHERY_ZONE': 'franklab.default',\n",
+       " 'FIGURL_CHANNEL': 'franklab2',\n",
+       " 'DJ_SUPPORT_FILEPATH_MANAGEMENT': 'TRUE',\n",
+       " 'KACHERY_CLOUD_EPHEMERAL': 'TRUE'}"
+      ]
+     },
+     "execution_count": 1,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
    "source": [
-    "from spyglass.settings import load_config\n",
+    "import os\n",
+    "import datajoint as dj\n",
+    "\n",
+    "if os.path.basename(os.getcwd()) == \"notebooks\":\n",
+    "    os.chdir(\"..\")\n",
+    "dj.config.load(\"dj_local_conf.json\")\n",
     "\n",
-    "load_config()"
+    "from spyglass.settings import config\n",
+    "\n",
+    "config"
    ]
   },
   {
@@ -260,10 +295,32 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 2,
    "id": "afb63913-4e6b-4049-ae1d-55ab1ac8d42c",
    "metadata": {},
    "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "[2023-09-28 08:07:06,176][INFO]: Connecting root@localhost:3307\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "[2023-09-28 08:07:06,254][INFO]: Connected root@localhost:3307\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Populate: Populating table DataAcquisitionDeviceSystem with data {'data_acquisition_device_system': 'SpikeGadgets'} using insert1.\n",
+      "Populate: Populating table DataAcquisitionDeviceAmplifier with data {'data_acquisition_device_amplifier': 'Intan'} using insert1.\n"
+     ]
+    },
     {
      "data": {
       "text/html": [
@@ -348,23 +405,11 @@
       "text/plain": [
        "*nwb_file_name nwb_file_a\n",
        "+------------+ +--------+\n",
-       "CH101_20210711 =BLOB=    \n",
-       "CH73_20211206_ =BLOB=    \n",
-       "CH65_20211212_ =BLOB=    \n",
-       "J1620210620_.n =BLOB=    \n",
-       "montague202008 =BLOB=    \n",
-       "chimi20200304_ =BLOB=    \n",
-       "Wallie20220913 =BLOB=    \n",
-       "mango20211203_ =BLOB=    \n",
-       "peanut20201108 =BLOB=    \n",
-       "wilbur20210406 =BLOB=    \n",
-       "eliot20221022_ =BLOB=    \n",
-       "Dan20211109_.n =BLOB=    \n",
-       "   ...\n",
-       " (Total: 817)"
+       "\n",
+       " (Total: 0)"
       ]
      },
-     "execution_count": 3,
+     "execution_count": 2,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -375,6 +420,14 @@
     "Nwbfile()"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "13fd64af",
+   "metadata": {},
+   "source": [
+    "# Up Next"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "c6850095",