Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Kachery fixes #918

Merged
merged 5 commits into from
Apr 11, 2024
Merged

Kachery fixes #918

merged 5 commits into from
Apr 11, 2024

Conversation

samuelbray32
Copy link
Collaborator

Description

  • Fixes Error when kachery sharing old analysis nwb files #914

    • In AnalysisNwbfile.get_abs_path, if an analysis file exists in the database, get the filepath from the datajoint filepath entry. Otherwise, define a new one in the same manner as before.
    • This ensures that files downloaded through kachery are placed in the location where datajoint expects them regardless of changes to our standard
    • Generalization of the previous solution to the change in files structure for analysis_nwb_files introduced in Analysis_file_path #303

  • Fixes Redundant deprecated Kachery tables in common_nwbfile #911

    • Removes the deprecated versions of AnalysisNwbfileKachery and NwbfileKachery in the common schema

Checklist:

  • This PR should be accompanied by a release: No
  • (If release) I have updated the CITATION.cff
  • I have updated the CHANGELOG.md
  • I have added/edited docs/notebooks to reflect the changes
  • Drop the AnalysisNwbfileKachery and NwbfileKachery in the common schema once approved (they're empty)

@edeno edeno requested a review from CBroz1 April 5, 2024 22:19
@edeno edeno requested review from CBroz1 and removed request for CBroz1 April 10, 2024 17:58
@edeno edeno merged commit d40424f into master Apr 11, 2024
3 checks passed
@edeno edeno deleted the kachery_fixes branch April 11, 2024 00:22
@samuelbray32 samuelbray32 mentioned this pull request Apr 11, 2024
Merged
edeno added a commit that referenced this pull request Apr 25, 2024
@CBroz1 @samuelbray32 @edeno
Part delete edits (#946)
2d9cc73 
* Add spyglass version to created analysis nwb files (#897)

* Add sg version to created analysis nwb files

* update changelog

* Change existing source script to spyglass version (#900)

* Add pynapple support (#898)

* Preliminary code

* Add retrieval of file names

* Add get_nwb_table function

* Update docstrings

* Update CHANGELOG.md

* Hot fixes for clusterless `get_ahead_behind_distance` and `get_spike_times` (#904)

* Squeeze results

* Make method and not class method

* Update CHANGELOG.md

* fix bugs in fetch_nwb (#913)

* Check for entry in merge part table prior to insert (#922)

* check for entry in merge part table prior to insert

* update changelog

* Kachery fixes (#918)

* Prioritize datajoint filepath for getting analysis file abs_path

* remove deprecated kachery tables

* update changelog

* fix lint

---------

Co-authored-by: Samuel Bray <samuelbray@som-dfvnn9m-lt.ucsf.edu>
Co-authored-by: Eric Denovellis <edeno@users.noreply.github.com>

* remove old tables from init (#925)

* Fix improper uses of strip (#929)

Strip will remove leading characters

* Update CHANGELOG.md

* Misc Issues (#903)

* #892

* #885

* #879

* Partial address of #860

* Update Changelog

* Partial solve of #886 - Ask import

* Fix failing tests

* Add note on order of inheritace

* #933

* Could not replicate fill_nan error. Reverting except clause

* Export logger (#875)

* WIP: rebase Export process

* WIP: revise doc

* ✅ : Generate working export script

* Cleanup: Expand notebook, migrate export process from graph class to export

* Revert dj_chains related edits

* Update changelog

* Revise doc

* Address review comments #875

* Remove walrus in  eval

* prevent log on preview

* Fix arg order on fetch, iterate over restr

* Add upstream analysis files during cascade. Address false positive fetch

* Avoid regen file list on revisit node

* Bump Export.Table.restr to mediumblob

* Revise Export.Table uniqueness to include export_id

* Spikesorting quality of life helpers (#910)

* add utitlity function for finding spikesorting merge ids

* add option to select v1 sorts that didn't go through artifact detection

* add option to return merge keys as dicts for future restrictions

* Add tool to get brain region and electrode info for a spikesorting merge id

* update changelog

* style cleanup

* style cleanup

* fix restriction bug for curation_id

* account for change or radiu_um argument name in spikeinterface

* only do joins with metric curastion tables if have relevant keys in the restriction

* Update tutorial to use spikesorting merge table helper functions

* fix spelling

* Add logging of AnalysisNwbfile creation time and file size (#937)

* Add logging for any func that creates AnalysisNwbfile

* Migrate create to top of respective funcs

* Use pathlib for file size. Bump creation time to top of  in spikesort

* Clear pre_create_time on create

* get/del -> pop

* Log when file accessed (#941)

* Add logging for any func that creates AnalysisNwbfile

* Fix bug on empty delete in merge table (#940)

* fix bug on empty delete in merge table

* update changelog

* fix spelling

---------

Co-authored-by: Chris Brozdowski <Chris.Broz@ucsf.edu>

* Remove master restriction

* Part delete takes restriction from self

---------

Co-authored-by: Samuel Bray <sam.bray@ucsf.edu>
Co-authored-by: Eric Denovellis <edeno@users.noreply.github.com>
Co-authored-by: Samuel Bray <samuelbray@som-dfvnn9m-lt.ucsf.edu>
Co-authored-by: Eric Denovellis <edeno@bu.edu>
edeno added a commit that referenced this pull request May 6, 2024
@samuelbray32 @CBroz1 @edeno
Group part delete propagation (#899)
435b0f3 
* Create class for group parts to help propagate deletes

* spelling

* update changelog

* Part delete edits (#946)

* Add spyglass version to created analysis nwb files (#897)

* Add sg version to created analysis nwb files

* update changelog

* Change existing source script to spyglass version (#900)

* Add pynapple support (#898)

* Preliminary code

* Add retrieval of file names

* Add get_nwb_table function

* Update docstrings

* Update CHANGELOG.md

* Hot fixes for clusterless `get_ahead_behind_distance` and `get_spike_times` (#904)

* Squeeze results

* Make method and not class method

* Update CHANGELOG.md

* fix bugs in fetch_nwb (#913)

* Check for entry in merge part table prior to insert (#922)

* check for entry in merge part table prior to insert

* update changelog

* Kachery fixes (#918)

* Prioritize datajoint filepath for getting analysis file abs_path

* remove deprecated kachery tables

* update changelog

* fix lint

---------

Co-authored-by: Samuel Bray <samuelbray@som-dfvnn9m-lt.ucsf.edu>
Co-authored-by: Eric Denovellis <edeno@users.noreply.github.com>

* remove old tables from init (#925)

* Fix improper uses of strip (#929)

Strip will remove leading characters

* Update CHANGELOG.md

* Misc Issues (#903)

* #892

* #885

* #879

* Partial address of #860

* Update Changelog

* Partial solve of #886 - Ask import

* Fix failing tests

* Add note on order of inheritace

* #933

* Could not replicate fill_nan error. Reverting except clause

* Export logger (#875)

* WIP: rebase Export process

* WIP: revise doc

* ✅ : Generate working export script

* Cleanup: Expand notebook, migrate export process from graph class to export

* Revert dj_chains related edits

* Update changelog

* Revise doc

* Address review comments #875

* Remove walrus in  eval

* prevent log on preview

* Fix arg order on fetch, iterate over restr

* Add upstream analysis files during cascade. Address false positive fetch

* Avoid regen file list on revisit node

* Bump Export.Table.restr to mediumblob

* Revise Export.Table uniqueness to include export_id

* Spikesorting quality of life helpers (#910)

* add utitlity function for finding spikesorting merge ids

* add option to select v1 sorts that didn't go through artifact detection

* add option to return merge keys as dicts for future restrictions

* Add tool to get brain region and electrode info for a spikesorting merge id

* update changelog

* style cleanup

* style cleanup

* fix restriction bug for curation_id

* account for change or radiu_um argument name in spikeinterface

* only do joins with metric curastion tables if have relevant keys in the restriction

* Update tutorial to use spikesorting merge table helper functions

* fix spelling

* Add logging of AnalysisNwbfile creation time and file size (#937)

* Add logging for any func that creates AnalysisNwbfile

* Migrate create to top of respective funcs

* Use pathlib for file size. Bump creation time to top of  in spikesort

* Clear pre_create_time on create

* get/del -> pop

* Log when file accessed (#941)

* Add logging for any func that creates AnalysisNwbfile

* Fix bug on empty delete in merge table (#940)

* fix bug on empty delete in merge table

* update changelog

* fix spelling

---------

Co-authored-by: Chris Brozdowski <Chris.Broz@ucsf.edu>

* Remove master restriction

* Part delete takes restriction from self

---------

Co-authored-by: Samuel Bray <sam.bray@ucsf.edu>
Co-authored-by: Eric Denovellis <edeno@users.noreply.github.com>
Co-authored-by: Samuel Bray <samuelbray@som-dfvnn9m-lt.ucsf.edu>
Co-authored-by: Eric Denovellis <edeno@bu.edu>

* Fix linting

---------

Co-authored-by: Chris Brozdowski <Chris.Broz@ucsf.edu>
Co-authored-by: Eric Denovellis <edeno@users.noreply.github.com>
Co-authored-by: Samuel Bray <samuelbray@som-dfvnn9m-lt.ucsf.edu>
Co-authored-by: Eric Denovellis <edeno@bu.edu>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@CBroz1 CBroz1 CBroz1 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Error when kachery sharing old analysis nwb files Redundant deprecated Kachery tables in common_nwbfile
3 participants
@samuelbray32 @CBroz1 @edeno