Adding support for irregular grids and unstructured data

[w-k-jones]

One of the long term goals for tobac is to add native support for detection, segmentation and tracking on data that is not on a cartesian grid, including high-resolution global model output in icosahedral/cubesphere grids, radar data in polar coordinates or even completely unstructured data such as a set of measurements from an array of ground stations.

I have started work on some functions to perform labelling/watershedding on arbitrary unstructured data: https://github.com/w-k-jones/unstructured_morphology. These work similar to the scipy.ndimage and scikit-learn labelling and watershedding functions, but instead take 1D arrays of data and coordinates and use a BallTree distanced based search to find neighbouring pixels. This approach is also adaptable to periodic boundaries and lat/lon tracking by using custom distance functions. The main downside is computational cost. While there is some room for optimisation, I expect at least a 20x slow-down compared to the existing functions.

What would be the best way to incorporate these into tobac? We could handle things behind the scenes, by looking for cases where multiple spatial coordinates correspond to 1 dimension, or make it an explicit requirement for the user to tell tobac when to use an unstructured approach.

[freemansw1]

Thanks for starting work on this @w-k-jones. An important topic, and it seems like you've made a lot of progress.

The computational slow-down is likely inevitable. I imagine the tree functionality is about as optimized as it reasonably can be given how well used those packages are (although that has never stopped bugs before).

For your specific question, I think we could go with a hybrid approach. Something like a unstructured_grid='auto' option for automatic detection, and then the explicit ability to turn on/off for the inevitable edge cases. That said, I'm far from an expert on unstructured grids (and have only worked with those data once), so would defer to an expert here.

A follow-up question for you. Would you aim to have your unstructured grid functions as separate from tobac (and as a dependency), or would you want to incorporate them? There are advantages and disadvantages either way.