Question on single camera/single estimator smoothing

[jpoberhauser]

Hello,

This is a general question around usage.

Thanks for releasing the eks code! I am wondering if there is a way to leverage a kalman smoother on a single camera, single estimator setting. So instead of aggregating different model predictions, if there is a way to use the spatially constrained kalman smoother to clean up the predictions on a single camera and single estimator ouputs?

Thank you in advance.

[themattinthehatt]

@jpoberhauser we haven't tried using EKS with a single estimator; the main motivation for using an ensemble is that we've found the confidence estimates for single pose estimators tend to not be calibrated super well.

That being said, you could certainly hack around the code a bit and see what happens. To do so you can start at this line, which computes the ensemble mean/median and ensemble variance:
https://github.com/paninski-lab/eks/blob/main/eks/singlecam_smoother.py#L48

You will need to replace the ensemble mean/median with the predictions from your single estimator. For the ensemble variance, you'll have to come up with a way of approximating this using the keypoint likelihood. For example, if the likelihood is 1, you can set the ensemble variance to 0; if the likelihood is 0, you can set the ensemble variance to some large value that makes sense in the context of your data (e.g. 400, or 1000; the scale is pixels^2). Then you can use the remainder of the ensemble_kalman_smoother_singlecam function as-is. You might need to fiddle around with how you map likelihood to "ensemble variance" but hopefully you get the idea.

Curious to hear how this works! (regardless of good or bad outcome)

[jpoberhauser]

Thank so much for your response @themattinthehatt, I really appreciate it and I'm a big fan of you all's work.

I will try that and I can report back. I am mostly looking for a way to add a fixed-interval smoother to predictions made in a single camera setting to try to offset the noise inherent in some pose estimators (jitter from frame to frame in videos).

Curious if your work with pose-lightning made these estimates less noisy since you use temporal smoothing?

Thanks again, appreciate the pointers.

[themattinthehatt]

Thanks for the kind words!

We found that the temporal loss in Lightning Pose actually didn't work so well unless we only penalized large jumps, so that loss doesn't contribute much to smoothing out small jitter. The context model I think does a better job at that since it can look at multiple frames to smooth over noise.

I'll also note that the ensemble_kalman_smoother_singlecam function is quite general, you can use it with DLC or SLEAP or LP or even a mixture of those networks. So it might be worth fitting a few more models with whichever package you're using and try the full EKS model as a baseline to compare against the single estimator EKS hack. Then you can at least have a better idea of what you're losing out on by not doing the full ensembling.

[jpoberhauser]

That all makes sense. Thanks again for the pointers. I will definitely try a couple of my models and see how far I can get with the ensemble smoother approach.