Select by bounding box

[kylebarron]

Building on top of #412 with help from @batpad

Change list

• Draw bbox in deck instead of separate layer

Notes

• Create mode for when picking bbox should be active
• Send data back to Python
• Make multiple draw calls, one per set of layer ids, then set on layer state
• Maybe delegate to sub models directly. Add a pick callback directly onto the models, and then iterate over each model, calling pick which picks and then assigns directly onto that model's state?
• In testing, we seemed to get only 35k rows back when we got 143k rows checking the same bbox in geopandas with the input gdf. This is possibly related to x,y,width, height passed in to pickObjects? Fixed with latest commit
• Note about crossing international dateline
• Ensure we're not creating more than 256 layers.

[batpad]

@kylebarron I added a "selectionMode" and some ugly UI for a user to start selection mode and clear selection. As a bonus, when the user has a selection, it also displays the number of objects selected.

There's a few things I feel a bit unsure about the logic, and also how the code is structured. We can look tomorrow - but I think this does roughly what we'd want.

[kylebarron]

It probably makes sense to keep the bounding box state on the widget model so that Python has access to the same bounding box. So e.g. connecting to Mosaic you could pass the bounding box for them to do the same query in duckdb

[batpad]

There would still a few things to be done here to finish, here's what it looks like so far:

BboxScreengrab.mp4

@kylebarron - Although, I'm wondering if this can be restructured to not be so tightly coupled to the main rendering code in index.tsx - it would be almost nice to be able to build something like this as a separate widget, not inside the core lonboard code-base. I think that's probably doable - it might be nice to talk through next week.

cc @hanbyul-here

[isaacbrodsky]

nit: this and the useState above can all be const

[kylebarron]

I think this is me writing too much Rust and always using let...

[isaacbrodsky]

Suggested change

	// Show the selection start point (note this does not show the proposed bounding box, but could be done)
	// Show the selection start point

Since a hover box is now shown

[batpad]

@kylebarron for now, this just sets map.selected_bounds after the user selects the bbox on the python side.

As an approach, I think I prefer this than trying to send indexes for all the objects down from JS - can we not implement this as a method in python and just use the bbox sent by the frontend? So a map.get_selected_objects() or so that would read the bounds from selected_bounds but then just do that query in Python to get the selected objects?

I feel like this is almost ready for review -- not sure what else we should do here.

@hanbyul-here would you be able to give this a look and point out anything obvious that you feel should be changed?

There's possibly a few things that could maybe be done nicer?

• The selection "mode" stuff seems a bit janky: https://github.com/developmentseed/lonboard/pull/417/files#diff-0b5adbfe7b36e4ae2f479291e20152e33e940f7f265162d77f40f6bdb5da7405R287
• The CSS to position the select control just seems terrible: https://github.com/developmentseed/lonboard/pull/417/files#diff-0b5adbfe7b36e4ae2f479291e20152e33e940f7f265162d77f40f6bdb5da7405R277

Also this was my first time really writing TypeScript so if there's anything there that seems like it's not proper..

One thing to note:

• Right now, "setting" map.selected_bounds in python will not do anything on the JS side - if we want a way for a user to programmatically set the selection in Python, I would rather do this by an explicit method on map like "set_selected_bounds", and treat it similar to how the flyTo operation is being handled currently.

[kylebarron]

As an approach, I think I prefer this than trying to send indexes for all the objects down from JS - can we not implement this as a method in python and just use the bbox sent by the frontend? So a map.get_selected_objects() or so that would read the bounds from selected_bounds but then just do that query in Python to get the selected objects?

I think there are pros and cons to each manner of implementation, but I think that there are some cases where maintaining the indexes on the model itself is valuable. It might also enable more client side use cases where you're syncing the data and the selection with another external widget through JS, avoiding the Python kernel.

And not needing to do another search query on the Python side might be nice for performance as well. And if we're sending up to like 100k uint32s encoded in Parquet to the server, that's a pretty small file. Probably smaller than 50kb?

[kylebarron]

Note that a user's selection on the screen is in pixels, and does not always align with an axis-aligned bounding box. E.g. if the map is tilted, the pixel bounding box will accurately choose items within that view because deck.gl uses the actual scene rendering for picking. But that won't be able to be picked from Python because you'd have to serialized the entire view and camera state and implement 3d picking from scratch.

[batpad]

@kylebarron based on our conversation, I've made it now so that each layer has a selected_bounds property and when a user selects a bbox, it will set that property on each layer on the map.

[vgeorge]

Hey folks, in #567 we fixed the bbox select drawing issues. Items moving forward:

• Merge main into this branch
• Add a toolbar: I believe we need two icons to implement this workflow, one to start a rectangle selection and another to clear an existing selection. This toolbar could also host other features in the future, like exporting image still or animations cc @LanesGood
• Add a panel to display the number of features selected
• Disable map drag when drawing the rectangle
• Update the cursor type when drawing
• Discuss what is the best approach to select the features; quoting my own comment for the other PR:

Deck.gl's pickObjects doesn't seem to pick all objects in the bbox. It might be because it doesn't pick occluded features. The function pickMultipleObjects seems to be the only method that picks occluded features, but the selection area must be a circle. If we find a way to get the feature IDs, we can implement the approach suggested by @kylebarron, but at the moment, the only approach seems to be passing the bbox to the Python code so it can query the features.

[vgeorge]

Status Update:

• Implemented XState for state management, replacing the reducer approach. This should simplify handling of side effects and reduce overall complexity
• Enhanced bounding box visuals:
  • Updated style during drawing phase
  • Changed cursor to pointer when drawing
• Integrated dotenv for environment variable management during frontend build. This was necessary for toggling the XState inspector

To Do:

• Add styles to toolbar button to initiate bounding box selection
• Verify correct transmission of bounds to Python core

@kylebarron @batpad following our recent discussion, bounds are being passed to the Python core in this section. But I'm unsure how to confirm this in the Jupyter notebook environment.

[kylebarron]

But I'm unsure how to confirm this in the Jupyter notebook environment.

You can add an attribute on the BaseArrowLayer that's

class BaseArrowLayer:
    selected_bounds = traitlets.Any().tag(sync=True)

Then after a making a selection on the map, you should be able to check layer.selected_bounds

[vgeorge]

@kylebarron @batpad this is ready for a review. I added a toolbar to handle the different steps of adding a bounding box and connected to the layer attributes selected_bounds property. Colors and icons are not super great, but I want to improve them in a separated ticket. Here is a screen capture of the current workflow:

[kylebarron]

This looks good at a glance to me, though I'd love @batpad to check it out as well if he has time!

[kylebarron]

This looks good at a glance to me, though I'd love @batpad to check it out as well if he has time!

[vgeorge]

@kylebarron @batpad thanks for the review, I'll merge this now. Sanjay and I had a quick chat about the XState implementation and we agreed some documentation about it would be helpful. I'll open a ticket to track it.