Enhance training dataset format support

[talmo]

Overview

Goal: Be able to store SLEAP package files (example) in PoseTraining objects so we can embed images used for training models together with the ground truth poses.

Data model equivalences

sleap-io	ndx-pose
Labels	PoseTraining
LabeledFrame	TrainingFrame
Skeleton	Skeleton
Instance	SkeletonInstance
Video	ImageSeries
Labels.skeletons	Skeletons
Labels.labeled_frames	TrainingFrames
Labels.videos	SourceVideos

TODO

MVP

• Make TrainingFrame.skeleton_instance into TrainingFrame.skeleton_instances so it can hold multiple instances per frame (Add SkeletonInstances in TrainingFrame talmolab/ndx-pose#3)
• Put SkeletonInstance creation inside of TrainingFrame loop in the README example
• Store image data in Image as 3-D array (Add tests for using Image in TrainingFrame.source_image talmolab/ndx-pose#2)
• Make TrainingFrame.source_video optional so we don't need to keep the source video together with the NWB file (Add tests for using Image in TrainingFrame.source_image talmolab/ndx-pose#2)
• Integrate into sleap_io.io.nwb
• Update docs/readme
• Finish tests

Nice to have

• Add a plain string field to TrainingFrame to point to source video as provenance but without triggering validation on DANDI
• Add support for PNG string encoding representation for Image
• Convert and upload reference SLEAP training datasets to DANDI
• Add support for directly opening SLEAP Dandisets in sleap-io via dandi API

[rly]

Hi @talmo and @eberrigan, thank you for all your work on this PR and for looking through the recent changes in the main branch at brainhack! I am happy to take on updating the README example and test, but I think you all would be better suited to do Integrate into sleap_io.io.nwb and Update docs/readme. Let me know if you'd like me to jump in to update the README example and test.

[eberrigan]

Feel free to update the readme. Thanks so much!

…

On Fri, Mar 8, 2024, 3:05 PM Ryan Ly ***@***.***> wrote: Hi @talmo <https://github.com/talmo> and @eberrigan <https://github.com/eberrigan>, thank you for all your work on this PR and for looking through the recent changes in the main branch at brainhack! I am happy to take on updating the README example and test, but I think you all would be better suited to do Integrate into sleap_io.io.nwb and Update docs/readme. Let me know if you'd like me to jump in to update the README example and test. — Reply to this email directly, view it on GitHub <#24 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AZOPO6WZEICIYKAD36M2HXTYXI723AVCNFSM6AAAAABD6CULCGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBWGU2DIOBYGY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[rly]

@talmo Can you give me write access to your fork? It will be easier than me creating a fork of your fork and us having to merge each fork in one by one...

[talmo]

@talmo Can you give me write access to your fork? It will be easier than me creating a fork of your fork and us having to merge each fork in one by one...

Done! https://github.com/talmolab/ndx-pose/invitations

[rly]

Thanks! I made a few changes - I moved the Skeletons type out of the PoseTraining container type because the PoseEstimation type now requires a Skeleton object and, for people who don't have (or want to include) training data with their pose estimates, it feels weird in my opinion, to require them to make a PoseTraining container just to put Skeletons into it.

I also updated the readme and tests and made some other minor adjustments.

@talmo I would like to merge this PR soon so that it will be easier for me to refactor the repo to use the latest ndx-template. Then you could start a new PR with any necessary adjustments for integration with SLEAP and any other changes you want to make (or you can make those changes here if you have time). Would that be all right?

[talmo]

Yes that works for me @rly!