Support pmtile based admin layers; COUNTRY=global #1417

gislawill · 2025-01-23T16:02:19Z

Description

This addresses #1404 by adding support PMTiles (vector tiles) for admin boundaries. This branch does not yet support running analysis (zonal stats) on these PMTiles. We will add support in a following PR that involves querying a S3-hosted geoparquet.

How to test the feature:

Open the surge deployment
Pan and scroll around loading the tiles
Try to find admin-level based functionality that I missed

Checklist - did you ...

Test your changes with

REACT_APP_COUNTRY=rbd yarn start
REACT_APP_COUNTRY=cambodia yarn start
REACT_APP_COUNTRY=mozambique yarn start
Add / update necessary tests?
Add / update outdated documentation?

Screenshot/video of feature:

https://www.loom.com/share/a7c80f04dbbb43a5b4cd68334886d088?sid=cda64d68-f6b3-4307-8b9e-3bf827a3faac

github-actions · 2025-01-23T16:27:38Z

Build succeeded and deployed at https://prism-1417.surge.sh
(hash f71adf6 deployed at 2025-03-05T00:00:02)

gislawill · 2025-02-05T03:44:07Z

Heads up @ericboucher and @wadhwamatic, this is ready for review. I believe all features but the Analysis Layers are functional.

I did run into some performance issues with the loadBoundaryRelations functionality that is required for the Alerts module. We've have over 20000 admin level 2 regions in the dataset and that much data caused a slowdown in start up time. Are any deployments still using the alerts module??

…ithub.com/WFP-VAM/prism-app into feature/support-pmtile-based-admin-layers

gislawill · 2025-02-05T03:52:06Z

frontend/src/components/MapView/Layers/BoundaryLayer/index.tsx

+      <Source
+        id={`source-${layer.id}`}
+        type="vector"
+        url={`pmtiles://${layer.path}`}
+      >


Note: we might get some performance benefits by rendering one source per PMTiles even when we're pulling multiple layers from the same file.

wadhwamatic · 2025-02-05T04:24:05Z

Are any deployments still using the alerts module??

@gislawill great! Look forward to digging into this later. No, there are no active deployments that are using alerts. But it's a feature that we need functioning in the next month for upcoming deployments.

gislawill · 2025-02-05T04:35:14Z

@wadhwamatic sounds good. @ericboucher it'd be helpful to get your thoughts on loadBoundaryRelations (in frontend/src/components/Common/BoundaryDropdown/utils.ts) and if there's any way it could be simplified while still supporting alerts. That function alone is taking about 22 seconds to run on my computer 😅

wadhwamatic · 2025-02-10T14:17:41Z

Hi @gislawill. The chart feature accessed through map interaction only works admin 0 in the surge deployment.

I see the request is missing the admin code for admin level 1 and 2 requests. For example:
https://api.earthobservation.vam.wfp.org/stats/admin?level=1&id_code=undefined&coverage=full&vam=rfh&start=2024-01-22&end=2025-01-21

To reproduce:

Choose layer: rainfall > rainfall amount > rainfall aggregate 10-day
Click on the map. In the tooltip windows, choose country and see that the chart loads correctly
Choose any level below national and see the chart error:

gislawill · 2025-02-19T06:24:04Z

The chart feature accessed through map interaction only works admin 0 in the surge deployment.

@wadhwamatic this is should now be fixed via the layer.json updates in 83b324c and 5153708

wadhwamatic · 2025-03-04T05:45:14Z

@wadhwamatic this is should now be fixed via the layer.json updates in 83b324c and 5153708

Hi @gislawill. Sorry, I missed this before. I see the request is correctly passing the admin codes now for each level, thanks! Something is currently broken on the chart API though and we're getting null results. I'm following up with Valentin on that.

gislawill · 2025-03-04T17:20:47Z

Something is currently broken on the chart API though and we're getting null results. I'm following up with Valentin on that.

Thanks @wadhwamatic. I am also seeing cors errors from the charts api (https://api.earthobservation.vam.wfp.org/stats/admin) both in surge and local environments

wadhwamatic · 2025-03-04T17:28:34Z

@gislawill - Valentin fixed the chart API issue and that functionality is now working as expected for this PR. 🙌🏼

The outstanding issues I see are:

The go to feature shows a list of locations (with some errors in that long list) but the interaction doesn't work
Labels in the charts tab are not working properly when comparing locations and periods (see screenshot). The label 'Global' is also incorrectly getting applied for all charts
The analysis feature is not implemented (known gap, but let's discuss a plan)

Sorry - here's the screenshot!

gislawill · 2025-03-04T17:32:01Z

@wadhwamatic, could you point me to the screenshot?

Labels in the charts tab are not working properly when comparing locations and periods (see screenshot). The label 'Global' is also incorrectly getting applied for all charts

gislawill · 2025-03-04T23:54:22Z

@wadhwamatic

The outstanding issues I see are:

The go to feature shows a list of locations (with some errors in that long list) but the interaction doesn't work

Labels in the charts tab are not working properly when comparing locations and periods (see screenshot). The label 'Global' is also incorrectly getting applied for all charts

The analysis feature is not implemented (known gap, but let's discuss a plan)

Issues 1 and 2 have been addressed and issue 3 is being addressed in this PR #1437

wadhwamatic

Looks good from my side Will. Over to @ericboucher for his review

wadhwamatic · 2025-03-11T18:30:23Z

@gislawill - there's an issue I've noticed while testing that I'm hoping we can improve. When a layer is loaded (10-day rainfall for example), boundaries temporarily disappear but then are reloaded and brought to the front, making the experience of loading layers feel clunky. Do you see any opportunities to improve this?

ericboucher · 2025-03-11T19:45:47Z

frontend/src/config/shared/layers.json

    "type": "boundary",
-    "path": "data/global/adm0_simplified.json",


should we keep this one for now? (until we have the analysis part figured out?)

ericboucher · 2025-03-11T19:49:53Z

@gislawill really nice work, I'm quite impressed / pleased.

Once thing I would recommend is to explicitely lock the analysis feature when pmtiles are used, a few options for @wadhwamatic

hide the analysis button completely
disable the analysis tooltip with an explanatory tooltip
disable the admin layer selection dropdown with an explanatory tooltip

ericboucher · 2025-03-11T19:52:24Z

frontend/src/components/MapView/Layers/BoundaryDropdown/BoundaryDropdownOptions.tsx

-      layerDataSelector(boundaryLayer.id),
-    ) as LayerData<BoundaryLayerProps> | undefined;
-    const { data } = boundaryLayerData || {};
+    const baseBoundaryLayer = useSelector(


There seems to be an empty level/selector in the goto dropdown and a few weird things. Maybe linked to the underlying data?

ericboucher · 2025-03-11T19:52:51Z

frontend/src/components/MapView/Layers/BoundaryDropdown/BoundaryDropdownOptions.tsx

+    const baseBoundaryLayer = useSelector(
+      layerDataSelector(boundaryLayers[0].id),
+    );
+    const { data: baseBoundaryLayerData } = baseBoundaryLayer || {};


Might be worth adding a few commends detailing what's related to pmtiles?

ericboucher · 2025-03-11T19:53:53Z

frontend/src/components/MapView/Layers/BoundaryDropdown/utils.ts

@@ -65,7 +65,7 @@ export function getAdminBoundaryTree(
      partialTree.children[fp[currentLevelCode]] ?? {
        adminCode: fp[currentLevelCode],
        key: fp[layer.adminLevelNames[level]],
-        label: fp[locationLevelNames[level]],
+        label: fp[locationLevelNames[level]] ?? '',


Should we process out empty lables?

ericboucher · 2025-03-11T19:55:44Z

frontend/src/components/MapView/Layers/BoundaryLayer/index.tsx

+    return undefined;
+  }, [layer.format]);
+
+  // This is causing a pretty massive performance hit. It seems to only be necessary for alerts.


TODO, let's put this PR in a mergeable state, which means no breaking changes for existing deployments. Similarly to analysis maybe we could disable alerts for pmtiles based deployments for now?

ericboucher · 2025-03-11T19:56:12Z

frontend/src/components/MapView/LeftPanel/ChartsPanel/index.tsx

+    admProps => {
+      if (!multiCountry) {
+        return country;
+      }
+
+      if (!admProps) {
+        return '';
+      }
+
+      if (adminLookup) {
+        return admProps[adminLookup] as string;
+      }
+
+      return (admProps.admin0Name as string) || '';
+    },


Maybe add a few doctrings?

ericboucher · 2025-03-11T19:57:39Z

frontend/src/utils/pmtiles-utils.ts

@@ -0,0 +1,24 @@
+import { Protocol, PMTiles } from 'pmtiles';


Add a longer docstring explaining what this is used for and also update the README to explain how to add pmtiles layers

gislawill added 3 commits January 22, 2025 17:27

WIP

2d4ebf5

Updating configurations

fcb6f9f

fix lint

65dd959

gislawill added 2 commits January 23, 2025 14:06

Making it smarter

48c8528

Handling charts

6ce0de1

gislawill changed the title ~~Draft: Support pmtile based admin layers; COUNTRY=global~~ Support pmtile based admin layers; COUNTRY=global Jan 24, 2025

gislawill requested review from wadhwamatic and ericboucher January 24, 2025 01:26

Handling multiple admin layers

7aedac7

gislawill linked an issue Feb 5, 2025 that may be closed by this pull request

[Feature Request]: make use of pmtiles for admin boundaries #1404

Open

Merge branch 'master' into feature/support-pmtile-based-admin-layers

19924a3

gislawill marked this pull request as ready for review February 5, 2025 03:38

gislawill added 3 commits February 4, 2025 21:47

Undo unnecessary update

3317485

Merge branch 'feature/support-pmtile-based-admin-layers' of https://g…

1679403

…ithub.com/WFP-VAM/prism-app into feature/support-pmtile-based-admin-layers

Remove unnecessary layer config

958a248

gislawill commented Feb 5, 2025

View reviewed changes

gislawill added 2 commits February 18, 2025 19:13

Fixing popup chart

83b324c

Adding levels support

5153708

gislawill added 2 commits March 4, 2025 15:32

fix chart title

d4cc0df

Fixing goto

241c3b0

gislawill added 2 commits March 4, 2025 16:36

Update Chart labels

25566ce

Fix goto

f71adf6

wadhwamatic approved these changes Mar 5, 2025

View reviewed changes

ericboucher reviewed Mar 11, 2025

View reviewed changes

ericboucher approved these changes Mar 11, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support pmtile based admin layers; COUNTRY=global #1417

Support pmtile based admin layers; COUNTRY=global #1417

gislawill commented Jan 23, 2025 •

edited

Loading

github-actions bot commented Jan 23, 2025 •

edited

Loading

gislawill commented Feb 5, 2025

gislawill Feb 5, 2025

wadhwamatic commented Feb 5, 2025

gislawill commented Feb 5, 2025

wadhwamatic commented Feb 10, 2025

gislawill commented Feb 19, 2025

wadhwamatic commented Mar 4, 2025

gislawill commented Mar 4, 2025

wadhwamatic commented Mar 4, 2025 •

edited

Loading

gislawill commented Mar 4, 2025

gislawill commented Mar 4, 2025

wadhwamatic left a comment

wadhwamatic commented Mar 11, 2025 •

edited

Loading

ericboucher Mar 11, 2025

ericboucher commented Mar 11, 2025

ericboucher Mar 11, 2025

ericboucher Mar 11, 2025

ericboucher Mar 11, 2025

ericboucher Mar 11, 2025

ericboucher Mar 11, 2025

ericboucher Mar 11, 2025

		"type": "boundary",
		"path": "data/global/adm0_simplified.json",

		@@ -0,0 +1,24 @@
		import { Protocol, PMTiles } from 'pmtiles';

Support pmtile based admin layers; COUNTRY=global #1417

Are you sure you want to change the base?

Support pmtile based admin layers; COUNTRY=global #1417

Conversation

gislawill commented Jan 23, 2025 • edited Loading

Description

How to test the feature:

Checklist - did you ...

Screenshot/video of feature:

github-actions bot commented Jan 23, 2025 • edited Loading

gislawill commented Feb 5, 2025

Choose a reason for hiding this comment

wadhwamatic commented Feb 5, 2025

gislawill commented Feb 5, 2025

wadhwamatic commented Feb 10, 2025

gislawill commented Feb 19, 2025

wadhwamatic commented Mar 4, 2025

gislawill commented Mar 4, 2025

wadhwamatic commented Mar 4, 2025 • edited Loading

gislawill commented Mar 4, 2025

gislawill commented Mar 4, 2025

wadhwamatic left a comment

Choose a reason for hiding this comment

wadhwamatic commented Mar 11, 2025 • edited Loading

Choose a reason for hiding this comment

ericboucher commented Mar 11, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gislawill commented Jan 23, 2025 •

edited

Loading

github-actions bot commented Jan 23, 2025 •

edited

Loading

wadhwamatic commented Mar 4, 2025 •

edited

Loading

wadhwamatic commented Mar 11, 2025 •

edited

Loading