Running DLC Pose Estimation without RawPosition

[rpswenson]

We'd like to be able to run sgp.DLCPoseEstimation().populate(pose_estimation_key) without needing to already have an entry in the sgc.RawPosition() table. As it is now, the pipeline tries to pull from raw position, leading to this error:

Error stack

IndexError                                Traceback (most recent call last)
Cell In[17], line 22
      8 pose_estimation_key = (
      9     sgp.DLCPoseEstimationSelection.insert_estimation_task(
     10         {
   (...)
     18     )
     19 )
     21 ##populate DLC Pose Estimation
---> 22 sgp.DLCPoseEstimation().populate(pose_estimation_key)
     24 ##start smooth interpolation
     25 si_params_name = "just_nan"

File ~/datajoint-python/datajoint/autopopulate.py:247, in AutoPopulate.populate(self, suppress_errors, return_exception_objects, reserve_jobs, order, limit, max_calls, display_progress, processes, make_kwargs, *restrictions)
    241 if processes == 1:
    242     for key in (
    243         tqdm(keys, desc=self.__class__.__name__)
    244         if display_progress
    245         else keys
    246     ):
--> 247         status = self._populate1(key, jobs, **populate_kwargs)
    248         if status is True:
    249             success_list.append(1)

File ~/datajoint-python/datajoint/autopopulate.py:314, in AutoPopulate._populate1(self, key, jobs, suppress_errors, return_exception_objects, make_kwargs)
    312 self.__class__._allow_insert = True
    313 try:
--> 314     make(dict(key), **(make_kwargs or {}))
    315 except (KeyboardInterrupt, SystemExit, Exception) as error:
    316     try:

File ~/spyglass/src/spyglass/position/v1/position_dlc_pose_estimation.py:245, in DLCPoseEstimation.make(self, key)
    235 logger.logger.info("getting raw position")
    236 interval_list_name = (
    237     convert_epoch_interval_name_to_position_interval_name(
    238         {
   (...)
    243     )
    244 )
--> 245 spatial_series = (
    246     RawPosition()
    247     & {**key, "interval_list_name": interval_list_name}
    248 ).fetch_nwb()[0]["raw_position"]
    249 _, _, _, video_time = get_video_path(key)
    250 pos_time = spatial_series.timestamps

IndexError: list index out of range

[edeno]

Does this nwb file not have position time stamps? @MichaelCoulter ?

[samuelbray32]

I think the issue is that it doesn't have spatial data corresponding to the video they're analyzing. Currently the code assumes there's already position data in the nwb and uses that to define timestamps and some other metadata. It's a known ToDo in the code to get this from the video object in the nwb instead.

I can take a stab at this, since I think it's a reasonable use case users in other labs will have too

[edeno]

I thought we still stashed the timestamps in the position file even if there's not position data.

[samuelbray32]

not positive about rec_to_nwb, but trodes_to_nwb only creates spatial series if position tracking files are found. The video object does contain timestamps even if no position tracking

[samuelbray32]

The information that seems to get used from the spatial series is:

• timestamps (contained in the
• reference_frame (where zero is w/r/t image)
• comments (we could probably skip including these)
• conversion (meters/pixel, there is a field for it in the video object but our conversion don't currently put information there)

In general all the infoirmation for DLC should be in the video object, and in cases where it wasn't supplied at nwb conversion we could have a way for the user to include it at the DLC stage or edit the nwb file

Tables that pull from RawPosition in DLC:

• DLCCentroid
• DLCOrient
• DLCPoseEstimation

[samuelbray32]

@rpswenson @MichaelCoulter, do you have a an example key entry where you hit the issue that I could use for testing?

[edeno]

@khl02007 and @zoldello might also have an example

[MichaelCoulter]

the nwbfile name is "BS2820231107_.nwb" we eventually run into the error when trying to populate DLCPoseEstimation

[rpswenson]

@samuelbray32
here's the pose estimation key:

{'nwb_file_name': 'BS2820231107_.nwb',
 'epoch': 2,
 'video_file_num': 1,
 'project_name': 'BS2820231107',
 'dlc_model_name': 'BS2820231107_tutorial_00',
 'dlc_model_params_name': 'default',
 'task_mode': 'trigger'}

[rpswenson]

The changes from the new PR seem to have fixed pose estimation populate, but I am running into a similar out of range error with sgp.DLCSmoothInterp().populate(si_key):

Cell In[12], line 41
     29 bodyparts = ["greenLED", "redLED_C"]
     30 sgp.DLCSmoothInterpSelection.insert(
     31     [
     32         {
   (...)
     39     skip_duplicates=True,
     40 )
---> 41 sgp.DLCSmoothInterp().populate(si_key)
     42 (
     43     sgp.DLCSmoothInterp() & {**si_key, "bodypart": bodyparts[0]}
     44 ).fetch1_dataframe().plot.scatter(x="x", y="y", s=1, figsize=(5, 5))
     46 ##smoothinterpcohort

File ~/datajoint-python/datajoint/autopopulate.py:247, in AutoPopulate.populate(self, suppress_errors, return_exception_objects, reserve_jobs, order, limit, max_calls, display_progress, processes, make_kwargs, *restrictions)
    241 if processes == 1:
    242     for key in (
    243         tqdm(keys, desc=self.__class__.__name__)
    244         if display_progress
    245         else keys
    246     ):
--> 247         status = self._populate1(key, jobs, **populate_kwargs)
    248         if status is True:
    249             success_list.append(1)

File ~/datajoint-python/datajoint/autopopulate.py:314, in AutoPopulate._populate1(self, key, jobs, suppress_errors, return_exception_objects, make_kwargs)
    312 self.__class__._allow_insert = True
    313 try:
--> 314     make(dict(key), **(make_kwargs or {}))
    315 except (KeyboardInterrupt, SystemExit, Exception) as error:
    316     try:

File ~/spyglass/src/spyglass/position/v1/position_dlc_position.py:179, in DLCSmoothInterp.make(self, key)
    177 # Get DLC output dataframe
    178 logger.logger.info("fetching Pose Estimation Dataframe")
--> 179 dlc_df = (DLCPoseEstimation.BodyPart() & key).fetch1_dataframe()
    180 dt = np.median(np.diff(dlc_df.index.to_numpy()))
    181 sampling_rate = 1 / dt

File ~/spyglass/src/spyglass/position/v1/position_dlc_pose_estimation.py:151, in DLCPoseEstimation.BodyPart.fetch1_dataframe(self)
    150 def fetch1_dataframe(self):
--> 151     nwb_data = self.fetch_nwb()[0]
    152     index = pd.Index(
    153         np.asarray(
    154             nwb_data["dlc_pose_estimation_position"]
   (...)
    158         name="time",
    159     )
    160     COLUMNS = [
    161         "video_frame_ind",
    162         "x",
    163         "y",
    164         "likelihood",
    165     ]

IndexError: list index out of range

si_key here is:

{'nwb_file_name': 'BS2820231107_.nwb',
 'epoch': 2,
 'video_file_num': 1,
 'project_name': 'BS2820231107',
 'dlc_model_name': 'BS2820231107_tutorial_00',
 'dlc_model_params_name': 'default'}

[samuelbray32]

This is actually a different error related to #961. It should probably be be cleaned up in this PR as well. I'll shift to a draft and add them in