Merge pull request #3544 from magda-io/issue/3487

#3487: Update magda to use ASGS 2021 and latest LGAs boundaries

CHANGES.md

@@ -1,5 +1,14 @@
 # CHANGELOG
 
+## v4.1.0
+
+- #3487: Update to use ASGS 2021 and latest LGAs (2023) boundaries
+- Added `indexed` field to region index & Bump region index version to `27`
+ - Migration instruction can be found from [the migration doc](./docs/docs/migration/4.1.0.md)
+- Adjusted & improved region config entry skip logic
+- Improved the region loader performance
+- Fixed: region loader failed to convert number type id property into a string
+
 ## v4.0.1
 
 - Add helm chart configuration option allows users to disable gateway auto-gzip response feature

docs/docs/migration/4.1.0.md

@@ -0,0 +1,217 @@
+## v4.1.0
+
+## Search Engine Region Index Upgrade
+
+We upgrade region index mapping version to `27` and upgrade region data to use ASGS 2021 and latest LGAs (2023) boundaries. During the upgrade, indexer will auto-recreate the region index. Depends on your database size, this might take some time (5-30 mins depends on resources allocated).
+
+> To build your own region files & MVT tile service, please refer to this repo: https://github.com/magda-io/magda-regions
+
+To avoid serving incomplete region data during the re-indexing process, you can manually set the region index version that search API uses to the existing versions via [helm chart options](https://github.com/magda-io/magda/blob/944ae887842b98c51698d567435003be2e9dbefd/deploy/helm/internal-charts/search-api/values.yaml#L29).
+
+Moreover, we also need to set the region mapping served from search API [Get Region Types
+](https://dev.magda.io/api/v0/apidocs/index.html#api-Search-GetV0SearchRegionTypes) endpoint to ensure it matches the old region index version we set.
+
+e.g. you can set the following in your helm deployment values file / config to make search api still queries the previous region index version (`26`) while indexer's creating the new indexes:
+
+```
+search-api:
+ # set region index version used to previous version: 26
+ regionsIndexVersion: 26
+ appConfig:
+ # Set to previous region mapping for regions in index version 26 or older
+ # You can find region mapping from the repo here: https://github.com/magda-io/magda/blob/main/magda-search-api/src/main/resources/regionMapping.json
+ regionMapping:
+ regionWmsMap:
+ STE:
+ layerName: FID_STE_2011_AUST
+ server: https://tiles.magda.io/FID_STE_2011_AUST/{z}/{x}/{y}.pbf
+ regionProp: STE_CODE11
+ aliases:
+ - ste_code
+ - ste_code_2011
+ - ste
+ digits: 1
+ description: States and Territories (STE)
+ regionIdsFile: data/regionids/region_map-FID_STE_2011_AUST_STE_CODE11.json
+ serverType: MVT
+ serverSubdomains: []
+ serverMinZoom: 0
+ serverMaxNativeZoom: 12
+ serverMaxZoom: 28
+ bbox:
+ - 96.81694140799998
+ - -43.74050960300003
+ - 159.10921900799997
+ - -9.142175976999999
+ nameProp: STE_NAME11
+ SA4:
+ layerName: FID_SA4_2011_AUST
+ server: https://tiles.magda.io/FID_SA4_2011_AUST/{z}/{x}/{y}.pbf
+ regionProp: SA4_CODE11
+ aliases:
+ - sa4_code_2011
+ - sa4_code
+ - sa4
+ digits: 3
+ description: Statistical Area Level 4 (SA4)
+ regionIdsFile: data/regionids/region_map-FID_SA4_2011_AUST_SA4_CODE11.json
+ serverType: MVT
+ serverSubdomains: []
+ serverMinZoom: 0
+ serverMaxNativeZoom: 12
+ serverMaxZoom: 28
+ bbox:
+ - 96.81694140799998
+ - -43.74050960300003
+ - 159.10921900799997
+ - -9.142175976999999
+ nameProp: SA4_NAME11
+ SA3:
+ layerName: FID_SA3_2011_AUST
+ server: https://tiles.magda.io/FID_SA3_2011_AUST/{z}/{x}/{y}.pbf
+ regionProp: SA3_CODE11
+ aliases:
+ - sa3_code_2011
+ - sa3_code
+ - sa3
+ digits: 5
+ description: Statistical Area Level 3 (SA3)
+ regionIdsFile: data/regionids/region_map-FID_SA3_2011_AUST_SA3_CODE11.json
+ serverType: MVT
+ serverSubdomains: []
+ serverMinZoom: 0
+ serverMaxNativeZoom: 12
+ serverMaxZoom: 28
+ bbox:
+ - 96.81694140799998
+ - -43.74050960300003
+ - 159.10921900799997
+ - -9.142175976999999
+ nameProp: SA3_NAME11
+ SA2:
+ layerName: FID_SA2_2011_AUST
+ server: https://tiles.magda.io/FID_SA2_2011_AUST/{z}/{x}/{y}.pbf
+ regionProp: SA2_MAIN11
+ aliases:
+ - sa2_code_2011
+ - sa2_code
+ - sa2
+ digits: 9
+ description: Statistical Area Level 2 (SA2)
+ regionIdsFile: data/regionids/region_map-FID_SA2_2011_AUST_SA2_MAIN11.json
+ serverType: MVT
+ serverSubdomains: []
+ serverMinZoom: 0
+ serverMaxNativeZoom: 12
+ serverMaxZoom: 28
+ bbox:
+ - 96.81694140799998
+ - -43.74050960300003
+ - 159.10921900799997
+ - -9.142175976999999
+ nameProp: SA2_NAME11
+ SA1:
+ layerName: FID_SA1_2011_AUST
+ server: https://tiles.magda.io/FID_SA1_2011_AUST/{z}/{x}/{y}.pbf
+ regionProp: SA1_MAIN11
+ aliases:
+ - sa1_code_2011
+ - sa1_maincode_2011
+ - sa1_code
+ - sa1
+ digits: 11
+ description: Statistical Area Level 1 (SA1)
+ regionIdsFile: data/regionids/region_map-FID_SA1_2011_AUST_SA1_MAIN11.json
+ serverType: MVT
+ serverSubdomains: []
+ serverMinZoom: 0
+ serverMaxNativeZoom: 12
+ serverMaxZoom: 28
+ bbox:
+ - 96.81694140799998
+ - -43.74050960300003
+ - 159.10921900799997
+ - -9.142175976999999
+ nameProp: SA2_NAME11
+ LGA:
+ layerName: FID_LGA_2015_AUST
+ server: https://tiles.magda.io/FID_LGA_2015_AUST/{z}/{x}/{y}.pbf
+ regionProp: LGA_CODE15
+ aliases:
+ - lga_code_2015
+ - lga_code
+ - lga
+ - lga_code_2014
+ - lga_code_2012
+ - lga_code_2010
+ digits: 5
+ description: Local Government Area (LGA)
+ regionIdsFile: data/regionids/region_map-FID_LGA_2015_AUST_LGA_CODE15.json
+ serverType: MVT
+ serverSubdomains: []
+ serverMinZoom: 0
+ serverMaxNativeZoom: 12
+ serverMaxZoom: 28
+ bbox:
+ - 96.81694140799998
+ - -43.740509602999985
+ - 159.10921900799997
+ - -9.142175976999999
+ nameProp: LGA_NAME15
+ POA:
+ layerName: FID_POA_2011_AUST
+ server: https://tiles.magda.io/FID_POA_2011_AUST/{z}/{x}/{y}.pbf
+ regionProp: POA_CODE
+ aliases:
+ - poa_2011
+ - postcode_2011
+ - poa
+ - poa_code
+ - poa_code_2011
+ - postcode
+ - postcode_2015
+ digits: 4
+ dataReplacements:
+ - - "^(?=\\d\\d\\d$)"
+ - '0'
+ description: Postal Area (POA)
+ regionIdsFile: data/regionids/region_map-FID_POA_2011_AUST_POA_CODE.json
+ serverType: MVT
+ serverSubdomains: []
+ serverMinZoom: 0
+ serverMaxNativeZoom: 12
+ serverMaxZoom: 28
+ bbox:
+ - 96.81694140799998
+ - -43.59821500299999
+ - 159.10921900799997
+ - -9.142175976999999
+ nameProp: POA_NAME
+ COM_ELB_ID_2016:
+ layerName: FID_COM20160509_ELB
+ server: https://tiles.magda.io/FID_COM20160509_ELB/{z}/{x}/{y}.pbf
+ regionProp: DIV_ID
+ aliases:
+ - divisionid
+ - com_elb_id_2016
+ - com_elb_id
+ - com_elb
+ digits: 3
+ description: Commonwealth Electoral District
+ regionIdsFile: data/regionids/region_map-FID_COM20160509_ELB_DIV_ID.json
+ serverType: MVT
+ serverSubdomains: []
+ serverMinZoom: 0
+ serverMaxNativeZoom: 12
+ serverMaxZoom: 28
+ bbox:
+ - 96.81676599999997
+ - -43.740509999999986
+ - 159.1092189999999
+ - -9.142175999999996
+ nameProp: SORTNAME
+```
+
+Once indexer fully completes the new region index creation / indexing, you can remove the above config from deployment config and deploy again to make search API uses the latest version index.
+
+You can check the indexing progress via Indexer module logs.

magda-int-test-ts/indexer-setup.conf

@@ -7,7 +7,7 @@ registry.registerForWebhooks = false
 # only enable / use one region file for speeding up the initialisation
 regionSources = {
  COUNTRY {
- url = "https://github.com/magda-io/magda-regions/releases/download/v1.0.0/country.geojson"
+ url = "https://github.com/magda-io/magda-regions/releases/download/v2.0.0/country.ndjson"
  idField = "id"
  nameField = "name"
  order = 9