Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Check for entry in merge part table prior to insert #922

Merged
merged 2 commits into from
Apr 11, 2024
Merged

Check for entry in merge part table prior to insert #922

merged 2 commits into from
Apr 11, 2024

Conversation

samuelbray32
Copy link
Collaborator

Description

Fixes #920

  • Prior change in hash function allows possible duplicate entries in a merge part table with different merge_ids
  • during _merge_insert, prior to generation of UUID, check for presence of source primary key as an entry to avoid generating duplicates.

Checklist:

  • This PR should be accompanied by a release: no
  • (If release) I have updated the CITATION.cff
  • I have updated the CHANGELOG.md
  • I have added/edited docs/notebooks to reflect the changes

@samuelbray32 samuelbray32 requested a review from CBroz1 April 10, 2024 22:47
@edeno edeno merged commit aa4ab8e into master Apr 11, 2024
7 checks passed
@edeno edeno deleted the merge_table_duplicates branch April 11, 2024 00:18
edeno added a commit that referenced this pull request Apr 25, 2024
@CBroz1 @samuelbray32 @edeno
Part delete edits (#946)
2d9cc73 
* Add spyglass version to created analysis nwb files (#897)

* Add sg version to created analysis nwb files

* update changelog

* Change existing source script to spyglass version (#900)

* Add pynapple support (#898)

* Preliminary code

* Add retrieval of file names

* Add get_nwb_table function

* Update docstrings

* Update CHANGELOG.md

* Hot fixes for clusterless `get_ahead_behind_distance` and `get_spike_times` (#904)

* Squeeze results

* Make method and not class method

* Update CHANGELOG.md

* fix bugs in fetch_nwb (#913)

* Check for entry in merge part table prior to insert (#922)

* check for entry in merge part table prior to insert

* update changelog

* Kachery fixes (#918)

* Prioritize datajoint filepath for getting analysis file abs_path

* remove deprecated kachery tables

* update changelog

* fix lint

---------

Co-authored-by: Samuel Bray <samuelbray@som-dfvnn9m-lt.ucsf.edu>
Co-authored-by: Eric Denovellis <edeno@users.noreply.github.com>

* remove old tables from init (#925)

* Fix improper uses of strip (#929)

Strip will remove leading characters

* Update CHANGELOG.md

* Misc Issues (#903)

* #892

* #885

* #879

* Partial address of #860

* Update Changelog

* Partial solve of #886 - Ask import

* Fix failing tests

* Add note on order of inheritace

* #933

* Could not replicate fill_nan error. Reverting except clause

* Export logger (#875)

* WIP: rebase Export process

* WIP: revise doc

* ✅ : Generate working export script

* Cleanup: Expand notebook, migrate export process from graph class to export

* Revert dj_chains related edits

* Update changelog

* Revise doc

* Address review comments #875

* Remove walrus in  eval

* prevent log on preview

* Fix arg order on fetch, iterate over restr

* Add upstream analysis files during cascade. Address false positive fetch

* Avoid regen file list on revisit node

* Bump Export.Table.restr to mediumblob

* Revise Export.Table uniqueness to include export_id

* Spikesorting quality of life helpers (#910)

* add utitlity function for finding spikesorting merge ids

* add option to select v1 sorts that didn't go through artifact detection

* add option to return merge keys as dicts for future restrictions

* Add tool to get brain region and electrode info for a spikesorting merge id

* update changelog

* style cleanup

* style cleanup

* fix restriction bug for curation_id

* account for change or radiu_um argument name in spikeinterface

* only do joins with metric curastion tables if have relevant keys in the restriction

* Update tutorial to use spikesorting merge table helper functions

* fix spelling

* Add logging of AnalysisNwbfile creation time and file size (#937)

* Add logging for any func that creates AnalysisNwbfile

* Migrate create to top of respective funcs

* Use pathlib for file size. Bump creation time to top of  in spikesort

* Clear pre_create_time on create

* get/del -> pop

* Log when file accessed (#941)

* Add logging for any func that creates AnalysisNwbfile

* Fix bug on empty delete in merge table (#940)

* fix bug on empty delete in merge table

* update changelog

* fix spelling

---------

Co-authored-by: Chris Brozdowski <Chris.Broz@ucsf.edu>

* Remove master restriction

* Part delete takes restriction from self

---------

Co-authored-by: Samuel Bray <sam.bray@ucsf.edu>
Co-authored-by: Eric Denovellis <edeno@users.noreply.github.com>
Co-authored-by: Samuel Bray <samuelbray@som-dfvnn9m-lt.ucsf.edu>
Co-authored-by: Eric Denovellis <edeno@bu.edu>
edeno added a commit that referenced this pull request May 6, 2024
@samuelbray32 @CBroz1 @edeno
Group part delete propagation (#899)
435b0f3 
* Create class for group parts to help propagate deletes

* spelling

* update changelog

* Part delete edits (#946)

* Add spyglass version to created analysis nwb files (#897)

* Add sg version to created analysis nwb files

* update changelog

* Change existing source script to spyglass version (#900)

* Add pynapple support (#898)

* Preliminary code

* Add retrieval of file names

* Add get_nwb_table function

* Update docstrings

* Update CHANGELOG.md

* Hot fixes for clusterless `get_ahead_behind_distance` and `get_spike_times` (#904)

* Squeeze results

* Make method and not class method

* Update CHANGELOG.md

* fix bugs in fetch_nwb (#913)

* Check for entry in merge part table prior to insert (#922)

* check for entry in merge part table prior to insert

* update changelog

* Kachery fixes (#918)

* Prioritize datajoint filepath for getting analysis file abs_path

* remove deprecated kachery tables

* update changelog

* fix lint

---------

Co-authored-by: Samuel Bray <samuelbray@som-dfvnn9m-lt.ucsf.edu>
Co-authored-by: Eric Denovellis <edeno@users.noreply.github.com>

* remove old tables from init (#925)

* Fix improper uses of strip (#929)

Strip will remove leading characters

* Update CHANGELOG.md

* Misc Issues (#903)

* #892

* #885

* #879

* Partial address of #860

* Update Changelog

* Partial solve of #886 - Ask import

* Fix failing tests

* Add note on order of inheritace

* #933

* Could not replicate fill_nan error. Reverting except clause

* Export logger (#875)

* WIP: rebase Export process

* WIP: revise doc

* ✅ : Generate working export script

* Cleanup: Expand notebook, migrate export process from graph class to export

* Revert dj_chains related edits

* Update changelog

* Revise doc

* Address review comments #875

* Remove walrus in  eval

* prevent log on preview

* Fix arg order on fetch, iterate over restr

* Add upstream analysis files during cascade. Address false positive fetch

* Avoid regen file list on revisit node

* Bump Export.Table.restr to mediumblob

* Revise Export.Table uniqueness to include export_id

* Spikesorting quality of life helpers (#910)

* add utitlity function for finding spikesorting merge ids

* add option to select v1 sorts that didn't go through artifact detection

* add option to return merge keys as dicts for future restrictions

* Add tool to get brain region and electrode info for a spikesorting merge id

* update changelog

* style cleanup

* style cleanup

* fix restriction bug for curation_id

* account for change or radiu_um argument name in spikeinterface

* only do joins with metric curastion tables if have relevant keys in the restriction

* Update tutorial to use spikesorting merge table helper functions

* fix spelling

* Add logging of AnalysisNwbfile creation time and file size (#937)

* Add logging for any func that creates AnalysisNwbfile

* Migrate create to top of respective funcs

* Use pathlib for file size. Bump creation time to top of  in spikesort

* Clear pre_create_time on create

* get/del -> pop

* Log when file accessed (#941)

* Add logging for any func that creates AnalysisNwbfile

* Fix bug on empty delete in merge table (#940)

* fix bug on empty delete in merge table

* update changelog

* fix spelling

---------

Co-authored-by: Chris Brozdowski <Chris.Broz@ucsf.edu>

* Remove master restriction

* Part delete takes restriction from self

---------

Co-authored-by: Samuel Bray <sam.bray@ucsf.edu>
Co-authored-by: Eric Denovellis <edeno@users.noreply.github.com>
Co-authored-by: Samuel Bray <samuelbray@som-dfvnn9m-lt.ucsf.edu>
Co-authored-by: Eric Denovellis <edeno@bu.edu>

* Fix linting

---------

Co-authored-by: Chris Brozdowski <Chris.Broz@ucsf.edu>
Co-authored-by: Eric Denovellis <edeno@users.noreply.github.com>
Co-authored-by: Samuel Bray <samuelbray@som-dfvnn9m-lt.ucsf.edu>
Co-authored-by: Eric Denovellis <edeno@bu.edu>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@CBroz1 CBroz1 CBroz1 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Duplicate entries created in Merge table
3 participants
@samuelbray32 @CBroz1 @edeno