Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multichannel Timeseries #418

Merged
merged 90 commits into from
Nov 20, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
90 commits
Select commit Hold shift + click to select a range
df987c3
init multichannel timeseries dir
droumis Aug 28, 2024
56c2098
add index thumbnail
droumis Aug 28, 2024
75ca6d0
set no data ingestion
droumis Aug 28, 2024
7cefaa7
lock project
droumis Aug 28, 2024
188470b
remove comments in ana proj file
droumis Aug 28, 2024
b6ea71d
Add table and reorganize the index notebook
droumis Aug 28, 2024
ab668d9
add data download to anaconda-project.yml
droumis Aug 28, 2024
2b3ad58
add assets
droumis Aug 28, 2024
8c15343
add the recommended workflow and update final served app cell
droumis Aug 28, 2024
1f87c52
rename and fix links
droumis Aug 29, 2024
2b26a17
rebuild project lock
droumis Aug 29, 2024
a36f0fe
init large workflow
droumis Aug 29, 2024
b7f9de8
set no data ingestion to false
droumis Aug 29, 2024
90b81b4
add test data
droumis Aug 29, 2024
55689fa
scale down thumbnail
droumis Aug 29, 2024
fd08b9a
tmp fix for CI, create data pyramid
droumis Aug 29, 2024
33de120
utilize a nwb lfp file for large pyramid workflow
droumis Aug 30, 2024
da74988
clear outputs
droumis Aug 30, 2024
84bc307
selectively import wget
droumis Aug 30, 2024
9c9372e
fix data path
droumis Aug 30, 2024
077ad0c
fix wget url
droumis Aug 30, 2024
d10f115
fix paths for wget
droumis Aug 30, 2024
382c572
output row instead of column to display in notebook
droumis Aug 30, 2024
6485585
reorganize resources and use panel row instead of column
droumis Aug 30, 2024
ed8b3bd
fix links from index table
droumis Aug 30, 2024
016bfdb
remove multi-chan hyphen
droumis Aug 30, 2024
76067c3
pin lower holonote and relock
droumis Sep 25, 2024
5326b35
update index with new image of multichan
droumis Sep 27, 2024
4d52faa
add new static previews of multichan app steps
droumis Sep 27, 2024
9cf0735
add narrative context for time range annotation extension
droumis Sep 27, 2024
e1cbe8b
change factors for data pyramid
droumis Sep 28, 2024
62a63d4
add scalebar extension
droumis Oct 7, 2024
b6feecf
relock
droumis Oct 7, 2024
74e37a3
remove small alternate workflow from index
droumis Oct 7, 2024
5b3fbf2
init streaming workflow
droumis Oct 26, 2024
43b8ac6
working stream tutorial
droumis Nov 6, 2024
b9b8d88
add mne and mne_lsl to project
droumis Nov 7, 2024
b20bc39
minor updates to main workflow
droumis Nov 7, 2024
a0d7bf2
streaming checkpoint
droumis Nov 7, 2024
d6dcad1
adding category and relocking
droumis Nov 7, 2024
ccec762
attempt to fix git diff when new project yaml
maximlt Nov 7, 2024
617d8cd
update project
droumis Nov 7, 2024
ed505d8
timestamp fix
droumis Nov 7, 2024
5080738
lock
droumis Nov 7, 2024
ec87504
remove numpy import and shorten buffer size
droumis Nov 7, 2024
3a8eaa7
add watermark and display last app
droumis Nov 7, 2024
b29d1f9
add static previews and vids
droumis Nov 8, 2024
08b33af
clear outputs
droumis Nov 9, 2024
cc428d5
clear outputs
droumis Nov 9, 2024
fe79215
remove video files and link to youtube
droumis Nov 12, 2024
0b7479f
set explicit page size
droumis Nov 12, 2024
45afe63
large app pyramid indicator
droumis Nov 12, 2024
2bd98f1
fix link
droumis Nov 12, 2024
b34a0a1
replace wget with pooch and use conda when possible
droumis Nov 15, 2024
9a605d7
link to annotation annotation docs
droumis Nov 16, 2024
a431014
format annotations_df
droumis Nov 16, 2024
53b380b
fix code formatting
droumis Nov 16, 2024
ddaf5fb
fix serving app text
droumis Nov 16, 2024
15d4126
Merge branch 'main' into multi-chan-ts
droumis Nov 16, 2024
93667d0
typos and missing import
droumis Nov 16, 2024
64a7d82
deploy correct app
droumis Nov 16, 2024
f8b1a32
convert not serialize
droumis Nov 16, 2024
8b34972
reduce size of images
droumis Nov 16, 2024
ad86f9b
clean up
droumis Nov 16, 2024
27a94e5
remove streaming
droumis Nov 16, 2024
8f2cae9
relock to fix mne
droumis Nov 16, 2024
3d5b233
add show to deploy command
droumis Nov 16, 2024
7c351ba
fix display of static images and media
droumis Nov 16, 2024
9317b63
fix display of static images and media
droumis Nov 16, 2024
eb37d1a
fix
droumis Nov 16, 2024
dd5fe23
block static images and add pyramid widget as extension in index
droumis Nov 18, 2024
2527755
try to add a page break after static float elements
droumis Nov 18, 2024
93cfc7c
add LFP data to proj downloads
droumis Nov 18, 2024
b21273d
auto min pin
droumis Nov 19, 2024
1d3af0b
relock and add lab command
droumis Nov 19, 2024
d6351c7
Remove 240927_multichan-checkpoint.png from remote repo
droumis Nov 19, 2024
31bb602
prepend_index and remove benchmarking
droumis Nov 19, 2024
e926946
sort imports
droumis Nov 19, 2024
06e3edb
improve wording
droumis Nov 19, 2024
93345af
Removed extensions heading
droumis Nov 19, 2024
fe5e8c9
updates
droumis Nov 19, 2024
332232f
fix spelling in large workflow header image
droumis Nov 19, 2024
f51491f
use context manager to open close nwb file
droumis Nov 19, 2024
d2ace07
use context manager to open close nwb file
droumis Nov 19, 2024
927c635
remove dropdown for details
droumis Nov 19, 2024
b1a294a
clean up
droumis Nov 19, 2024
2dd5ede
fix large wf thumnails
droumis Nov 19, 2024
17a425f
shorten card description
droumis Nov 19, 2024
4485c02
close file later
maximlt Nov 20, 2024
c8120dd
update published date
maximlt Nov 20, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1,149 changes: 1,149 additions & 0 deletions multichannel_timeseries/0_multichan.ipynb

Large diffs are not rendered by default.

934 changes: 934 additions & 0 deletions multichannel_timeseries/1_large_multichan.ipynb

Large diffs are not rendered by default.

852 changes: 852 additions & 0 deletions multichannel_timeseries/anaconda-project-lock.yml

Large diffs are not rendered by default.

89 changes: 89 additions & 0 deletions multichannel_timeseries/anaconda-project.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
name: multichannel_timeseries
description: Display and annotate multichannel timeseries data in Neuroscience

examples_config:
created: 2024-11-20
maintainers:
- "droumis"
labels:
droumis marked this conversation as resolved.
Show resolved Hide resolved
- "hvplot"
- "panel"
- "holoviews"
- "datashader"
- "holonote"
categories:
droumis marked this conversation as resolved.
Show resolved Hide resolved
- Neuroscience
title: "Multichannel Timeseries"
deployments:
- command: notebook
- command: dashboard
resource_profile: default
auto_deploy: true
notebooks_to_skip: []
skip_notebooks_evaluation: false
no_data_ingestion: false
gh_runner: "ubuntu-latest"
droumis marked this conversation as resolved.
Show resolved Hide resolved

user_fields: [examples_config]

channels:
- conda-forge
- nodefaults

packages: &pkgs
- notebook>=6.5.2,<7 # required
- python=3.11
- numpy>=2.0.2 # auto min pinned 2024-11-18
- panel>=1.4.2
- hvplot>=0.10.0
- pandas>=2.2.1
- holoviews>=1.19.0 # 1.19: wide df handling, scalebars
- datashader>=0.16.3 # auto min pinned 2024-11-18
- xarray>=2024.5.0,<2024.10.0 # datatree merged into xarray 2024.10.0
- zarr>=2.18.3 # auto min pinned 2024-11-18
- pyarrow>=18.0.0 # auto min pinned 2024-11-18
- dask>=2024.11.2 # auto min pinned 2024-11-18
- h5py>=3.12.1 # auto min pinned 2024-11-18
- mne-lsl>=1.6.1 # auto min pinned 2024-11-18
- jupyterlab>=4.3.0 # auto min pinned 2024-11-18
- holonote>=0.2.1 # 0.2.1: visibility and coloring fixes
- pyproj==3.6.1
- bokeh>=3.6.0 # 3.6.0: scalebar on subplots
- ndpyramid==0.2.0 # avoid HybridMappingProxy error in 0.3.1
- pooch>=1.8.2 # auto min pinned 2024-11-18
- pip
- pip:
- tsdownsample>=0.1.3
- mne[hdf5]>=1.8.0 # auto min pinned 2024-11-18 # pip instead of conda to avoid missing io module error

dependencies: *pkgs

commands:
lab:
unix: jupyter lab index.ipynb
windows: jupyter lab index.ipynb
notebook:
notebook: index.ipynb
dashboard:
unix: panel serve --rest-session-info --session-history -1 multichan.ipynb --show
supports_http_options: true

variables: {}

downloads:
EEG_data:
url: https://datasets.holoviz.org/eeg/v1/S001R04.edf
description: |
EEG Dataset
filename: data/S001R04.edf
LFP_data:
url: https://datasets.holoviz.org/lfp/v1/sub-719828686_ses-754312389_probe-756781563_ecephys.nwb
description: |
Ephys NWB Dataset https://datasets.holoviz.org/lfp/README.md
filename: data/sub-719828686_ses-754312389_probe-756781563_ecephys.nwb

platforms:
- linux-64
- osx-64
- win-64
- osx-arm64
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
143 changes: 143 additions & 0 deletions multichannel_timeseries/index.ipynb
Original file line number Diff line number Diff line change
@@ -0,0 +1,143 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Multichannel Timeseries\n",
"\n",
"## Introduction\n",
"\n",
"Visualizing time series from various sources on a vertically stacked, time-aligned display is often the first tool employed when working with data from [electrophysiological](https://en.wikipedia.org/wiki/Electrophysiology) studies. These experiments generally seek to provide insight into the activities of nerve cells or muscles, as well as how they relate to each other or other measurable variables, such as the spatial position of the organism under study. Electrophysiological recording sessions can include diverse data types like electromyograms (EMG), electroencephalograms (EEG), local field potentials (LFP), or neural action potentials (spikes) - each consisting of multiple streams of information ('channels') that all are unified by their alignment to a single series of timestamps, but having a heterogenuous range of amplitude values.\n",
"\n",
"There are many different approaches for a visualization of multichannel timeseries data, but we'll highlight the one that we've found to be promising in many scenarios. If you have a dataset that is too large to fit into memory, check out the alternate approaches in the 'Extensions and Alternate Workflows' below the Recommended Workflow."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"___"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Key Features\n",
"Analyzing electrophysiological data often involves searching for patterns across time, channels, and covariates. Target features that support this type of investigation for time-aligned, amplitude-diverse data include:\n",
"\n",
"- **Smooth Interactions at Scale:** Smooth zooming and panning across time and channels.\n",
"- **Subcoordinate Axes:** Independent amplitude dimension (y-axis) per channel.\n",
"- **Instant Inspection:** Quick information preview about the data under the cursor.\n",
"- **Group-Aware Handling:** Zooming and y-range normalization per specified channel group/type.\n",
"- **Reference View:** Minimap for navigation and contextualization in large datasets.\n",
"- **Responsive Scale Bar:** Dynamic amplitude reference measurement.\n",
"- **Time-Range Annotations:** Create and edit time-range annotations directly on the plot."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 👉 [Recommended Workflow](./0_multichan.ipynb) (Start Here)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<img src='./assets/240927_multichan.png' alt=\"Multichannel timeseries recommended workflows\" align=\"right\" width=75%>\n",
"\n",
"The [recommended multichannel timeseries notebook](./0_multichan.ipynb) provides a workflow for processing and analyzing multichannel timeseries **datasets that fit in memory**.\n",
"\n",
"It covers the importance of live downsampling using algorithms that maintain the signal shape and optimize browser performance.\n",
"\n",
"Although it features the MNE-Python library for handling and visualization of raw EEG data, the concepts and methods are applicable to various timeseries data beyond EEG.\n",
"\n",
"<div style=\"clear: both;\"></div>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Extensions and Alternate Workflows"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"| Title | Preview | Type | Summary | Details |\n",
"|---|---|---|---|---|\n",
"| [Minimap Widget](./0_multichan.ipynb#Minimap-Extension) | <img src=\"./thumbnails/minimap.png\" alt=\"Minimap\" width=\"500\"> | Extension | Use HoloViews RangeToolLink and Datashader to rasterize an aggregate view. | Create a minimap widget that provides a condensed overview of the entire dataset, allowing users to select and zoom into areas of interest quickly in the main plot while maintaining the contextualization of the zoomed out view. |\n",
"| [Time Range Annotation](./0_multichan.ipynb#Time-Range-Annotation-Extension) | <img src=\"./thumbnails/annotation.png\" alt=\"annotation\" width=\"500\"> | Extension | Utilize HoloNote along with any primary workflow approach. | Create (or import), edit, and save a table of start and end times. View the categorized ranges overlaid on the multichannel timeseries plot. HoloNote allows you to interact with time range annotations directly on a plot, through widgets, or programmatically. |\n",
"| [Standalone App](./0_multichan.ipynb#Standalone-App-Extension) | <img src=\"./thumbnails/standalone.png\" alt=\"Standalone App\" width=\"500\"> | Extension | Deploy the visualization as a standalone, template-styled web application using HoloViz Panel. | This extension shows how to wrap your plot within a Panel Template for a styled, interactive web app that can run outside of a Jupyter Notebook. By marking the Panel component as servable, you can launch the app directly from the command line, providing a user-friendly interface in its own browser window. |\n",
"| [Scale Bar](./0_multichan.ipynb#scale-bar-extension) | <img src=\"./thumbnails/scalebar.png\" alt=\"Scale Bar\" width=\"500\"> | Extension | Provides an accurate, dynamic, and customizable reference gauge of signal amplitude. | The scale bar feature in HoloViews+Bokeh allows for precise measurement indicators on plots. Requires Bokeh>=3.6.0 and HoloViews>=1.19.2 |\n",
"| **[Larger Dataset <br> (> RAM)](./1_large_multichan.ipynb)** | <img src=\"./thumbnails/large_multichan.png\" alt=\"Larger Dataset\" width=\"500\"> | **Alternate Workflow** | Utilize Xarray, Zarr, and Dask for dynamic access of data subsets at optimal resolution. | To handle datasets beyond available memory (RAM), we can utilize dynamic access of certain data ranges and resolutions, using a precomputed hierarchical array pyramid. |\n",
"| [Pyramid Level Widget](./1_large_multichan.ipynb#Pyramid-Indicator-Extesion) | <img src=\"./thumbnails/pyramid_level.png\" alt=\"Pyramid Level\" width=\"500\"> | Extension | Indicator for the active Pyramid Level | Use a custom HoloViews Stream to update the indicator styling of the different resolution levels of the data | \n",
" "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Funding:\n",
"\n",
"- 2023 - 2024: Chan Zuckerberg Initiative. Learn more in the [grant announcement](https://blog.bokeh.org/announcing-czi-funding-for-bokeh-for-bioscience-5f74426c011a).\n",
" - Project Lead: Demetris Roumis, PhD (@droumis on [HoloViz Discord](https://discord.gg/X6Eq9CvZZn))\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.10"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added multichannel_timeseries/thumbnails/index.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added multichannel_timeseries/thumbnails/minimap.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added multichannel_timeseries/thumbnails/scalebar.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added multichannel_timeseries/thumbnails/streaming.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
10,854 changes: 10,854 additions & 0 deletions test_data/multichannel_timeseries/S001R04.edf

Large diffs are not rendered by default.

Loading