Merge pull request #1571 from cityofaustin/18365-jc-sig-eng-areas

Add new geospatial layers, trigger-managed attribute columns, and a fancy new AGOL helper

database/README.md

@@ -28,6 +28,7 @@ The design supports an editing environment which enables Vision Zero program sta
   - [Add a custom lookup value to a CRIS-managed lookup table (todo)](#add-a-custom-lookup-value-to-a-cris-managed-lookup-table-todo)
   - [Debugging record triggers](#debugging-record-triggers)
   - [Parsing change log data](#parsing-change-log-data)
+  - [Creating a new geospatial layer](#creating-a-new-geospatial-layer)
 - [Backups](#backups)
 - [Hasura](#hasura)
 - [Development and deployment](#development-and-deployment)
@@ -235,13 +236,18 @@ The view `crashes_change_log_view` provides a unioned view of the unified table
 
 We have a number of tables which function as geospatial layers which are referenced by crashes and various other records. At the Vision Zero team's request, our team is actively working to expand the number of layers available in the database as well as add new attribute columns to crash records which will be populated based on their intersection with these layers.
 
-| Table                 | Geometry type | description                                                                                               | owner/source                                                         |
-| --------------------- | ------------- | --------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- |
-| `council_districts`   | polygon       | City of Austin council districts                                                                          | ArcGIS Online authoritative layer owned by CTM GIS                   |
-| `atd_jurisdictions`   | polygon       | City of Austin jurisdictions                                                                              | ArcGIS Online authoritative layer owned by CTM GIS                   |
-| `engineering_areas`   | polygon       | TPW traffic engineering areas                                                                             | ArcGIS Online authoritative layer owned by DTS GIS                   |
-| `non_coa_roadways`    | polygon       | Polygon layer covering roadways which are not maintained by the City of Austin                            | ArcGIS Online authoritative layer maintained by Vision Zero GIS team |
-| `atd_txdot_locations` | polygon       | Aka, "location polygons", these shapes are used to group crashes based on an intersection or road segment | ArcGIS Online authoritative layer maintained by Vision Zero GIS team |
+See also the guidance for creating a new geospatial layer in the common maintance tasks section, below.
+
+| Table                   | Geometry type  | description                                                                                                      | owner/source                                                         |
+| ----------------------- | -------------- | ---------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- |
+| `council_districts`     | `MultiPolygon` | City of Austin council districts                                                                                 | ArcGIS Online authoritative layer owned by CTM GIS                   |
+| `atd_jurisdictions`     | `MultiPolygon` | City of Austin jurisdictions                                                                                     | ArcGIS Online authoritative layer owned by CTM GIS                   |
+| `engineering_areas`     | `MultiPolygon` | TPW traffic engineering areas                                                                                    | ArcGIS Online authoritative layer owned by DTS GIS                   |
+| `non_coa_roadways`      | `MultiPolygon` | Polygon layer covering roadways which are not maintained by the City of Austin                                   | ArcGIS Online authoritative layer maintained by Vision Zero GIS team |
+| `atd_txdot_locations`   | `MultiPolygon` | Aka, "location polygons", these shapes are used to group crashes based on an intersection or road segment        | ArcGIS Online authoritative layer maintained by Vision Zero GIS team |
+| `signal_engineer_areas` | `MultiPolygon` | Polygon zones assigned to traffic signal engineers                                                               | ArcGIS Online authoritative layer owned by DTS GIS                   |
+| `zip_codes`             | `MultiPolygon` | Polygons which represent the Zone Improvement Plan (ZIP) postal code areas in the Austin metro area              | ArcGIS Online authoritative layer owned by DTS GIS                   |
+| `apd_sectors`           | `MultiPolygon` | Polygons which represent Austin Police Department (APD) sectors and districts used for dispatching and reporting | ArcGIS Online authoritative layer owned by APD                       |
 
 ## Common maintenance tasks
 
@@ -351,6 +357,14 @@ WHERE
 LIMIT 10000;
 ```
 
+### Creating a new geospatial layer
+
+When creating a new database table to hold feature data, polygon geometries should always be stored in a `Multipolygon` column type to avoid future issues. Any columns that will be populated with AGOL feature attribute data should exactly match the column names used in AGOL, except they should be lowercase.
+
+As a best practice, tables should always be configured with `created_at` of type `timestamptz` with default `now()`. Layers should make use of the [ArcGIS Online Layer Helper](/toolbox/load_agol_layer) tool so that they can be easily refreshed.
+
+Typically, any foreign key constraint that references the layer should use the `ON UPDATE SET NULL` directive to ensure that the rows in the layer table can be deleted and re-inserted without being blocked by foreign references.
+
 ## Backups
 
 Daily database backups are managed via AWS RDS. Backups are retained for 14 Days.

database/metadata/databases/default/tables/geo_apd_sectors.yaml

@@ -0,0 +1,3 @@
+table:
+  name: apd_sectors
+  schema: geo

database/metadata/databases/default/tables/public_atd_jurisdictions.yaml → database/metadata/databases/default/tables/geo_atd_jurisdictions.yaml

@@ -1,3 +1,3 @@
 table:
   name: atd_jurisdictions
-  schema: public
+  schema: geo

database/metadata/databases/default/tables/public_council_districts.yaml → database/metadata/databases/default/tables/geo_council_districts.yaml

@@ -1,3 +1,3 @@
 table:
   name: council_districts
-  schema: public
+  schema: geo

database/metadata/databases/default/tables/public_engineering_areas.yaml → database/metadata/databases/default/tables/geo_engineering_areas.yaml

@@ -1,31 +1,31 @@
 table:
   name: engineering_areas
-  schema: public
+  schema: geo
 select_permissions:
   - role: editor
     permission:
       columns:
-        - area_id
-        - label
-        - updated_at
+        - engineering_area_id
+        - atd_engineer_areas
+        - created_at
         - geometry
       filter: {}
     comment: ""
   - role: readonly
     permission:
       columns:
-        - area_id
-        - label
-        - updated_at
+        - engineering_area_id
+        - atd_engineer_areas
+        - created_at
         - geometry
       filter: {}
     comment: ""
   - role: vz-admin
     permission:
       columns:
-        - area_id
-        - label
-        - updated_at
+        - engineering_area_id
+        - atd_engineer_areas
+        - created_at
         - geometry
       filter: {}
     comment: ""

database/metadata/databases/default/tables/public_non_coa_roadways.yaml → database/metadata/databases/default/tables/geo_non_coa_roadways.yaml

@@ -1,3 +1,3 @@
 table:
   name: non_coa_roadways
-  schema: public
+  schema: geo

database/metadata/databases/default/tables/geo_signal_engineer_areas.yaml

@@ -0,0 +1,3 @@
+table:
+  name: signal_engineer_areas
+  schema: geo

database/metadata/databases/default/tables/geo_zip_codes.yaml

@@ -0,0 +1,3 @@
+table:
+  name: zip_codes
+  schema: geo

database/metadata/databases/default/tables/public_fatalities_view.yaml

@@ -15,11 +15,11 @@ object_relationships:
     using:
       manual_configuration:
         column_mapping:
-          engineering_area_id: area_id
+          engineering_area_id: engineering_area_id
         insertion_order: null
         remote_table:
           name: engineering_areas
-          schema: public
+          schema: geo
   - name: recommendation
     using:
       manual_configuration:

database/metadata/databases/default/tables/tables.yaml

@@ -1,3 +1,10 @@
+- "!include geo_apd_sectors.yaml"
+- "!include geo_atd_jurisdictions.yaml"
+- "!include geo_council_districts.yaml"
+- "!include geo_engineering_areas.yaml"
+- "!include geo_non_coa_roadways.yaml"
+- "!include geo_signal_engineer_areas.yaml"
+- "!include geo_zip_codes.yaml"
 - "!include lookups_agency.yaml"
 - "!include lookups_autonomous_level_engaged.yaml"
 - "!include lookups_autonomous_unit.yaml"
@@ -46,7 +53,6 @@
 - "!include public_atd__coordination_partners_lkp.yaml"
 - "!include public_atd__recommendation_status_lkp.yaml"
 - "!include public_atd_apd_blueform.yaml"
-- "!include public_atd_jurisdictions.yaml"
 - "!include public_atd_txdot_locations.yaml"
 - "!include public_change_log_crashes.yaml"
 - "!include public_change_log_crashes_cris.yaml"
@@ -55,20 +61,17 @@
 - "!include public_change_log_units.yaml"
 - "!include public_change_log_units_cris.yaml"
 - "!include public_charges_cris.yaml"
-- "!include public_council_districts.yaml"
 - "!include public_crash_injury_metrics_view.yaml"
 - "!include public_crash_notes.yaml"
 - "!include public_crashes.yaml"
 - "!include public_crashes_change_log_view.yaml"
 - "!include public_crashes_cris.yaml"
 - "!include public_crashes_list_view.yaml"
 - "!include public_ems__incidents.yaml"
-- "!include public_engineering_areas.yaml"
 - "!include public_fatalities_view.yaml"
 - "!include public_location_crashes_view.yaml"
 - "!include public_location_notes.yaml"
 - "!include public_locations_list_view.yaml"
-- "!include public_non_coa_roadways.yaml"
 - "!include public_people.yaml"
 - "!include public_people_cris.yaml"
 - "!include public_people_list_view.yaml"

database/migrations/default/1729529574000_signal_eng_areas/down.sql

@@ -0,0 +1,3 @@
+drop table public.signal_engineer_areas;
+
+alter table non_coa_roadways add column id serial primary key;

database/migrations/default/1729529574000_signal_eng_areas/up.sql

@@ -0,0 +1,12 @@
+alter table non_coa_roadways drop column id;
+
+create table public.signal_engineer_areas (
+    signal_engineer_area_id integer unique not null,
+    signal_eng_area text not null,
+    geometry public.geometry (multipolygon, 4326) not null,
+    created_at timestamptz default now() not null
+);
+
+create index on signal_engineer_areas using gist (geometry);
+
+comment on table public.signal_engineer_areas is 'Polygon zones assigned to traffic signal engineers. These zones cover the Full and Limited Purpose juristdiction areas of the City of Austin.';

database/migrations/default/1729529575000_engineering_areas_edits/down.sql

@@ -0,0 +1,91 @@
+--revert to name updated_at
+alter table engineering_areas rename column created_at to updated_at;
+--revert time to timestamp with out time zone
+alter table engineering_areas alter column updated_at set data type timestamp using updated_at::timestamp;
+-- rever other col names
+alter table engineering_areas rename column engineering_area_id to area_id;
+alter table engineering_areas rename column atd_engineer_areas to label;
+
+-- revert to `area_id` column
+CREATE OR REPLACE FUNCTION public.crashes_set_spatial_attributes()
+ RETURNS trigger
+ LANGUAGE plpgsql
+AS $function$
+begin
+    if (new.latitude is not null and new.longitude is not null) then
+        -- save lat/lon into geometry col
+        new.position = st_setsrid(st_makepoint(new.longitude, new.latitude), 4326);
+        --
+        -- get location polygon id
+        --
+        if (new.rpt_road_part_id != 2 and upper(ltrim(new.rpt_hwy_num)) in ('35', '183','183A','1','290','71','360','620','45','130')) then
+            -- use level 5 polygon
+            new.location_id = (
+                select
+                    location_id
+                from
+                    public.atd_txdot_locations
+                where
+                    location_group = 2 -- level 5
+                    and st_contains(geometry, new.position)
+                limit 1);
+        else
+            -- use the other polygons
+            new.location_id = (
+                select
+                    location_id
+                from
+                    public.atd_txdot_locations
+                where
+                    location_group = 1 -- not level 5
+                    and st_contains(geometry, new.position)
+                limit 1);
+        end if;
+
+        raise debug 'found location: % compared to previous location: %', new.location_id, old.location_id;
+        --
+        -- check if in austin full purpose jurisdiction
+        --
+        new.in_austin_full_purpose =  st_contains((select geometry from atd_jurisdictions where id = 5), new.position);
+        raise debug 'in austin full purpose: % compared to previous: %', new.in_austin_full_purpose, old.in_austin_full_purpose;
+        --
+        -- get council district
+        --
+        new.council_district = (
+            select
+                council_district
+            from
+                public.council_districts
+            where
+                st_contains(geometry, new.position)
+            limit 1);
+        raise debug 'council_district: % compared to previous: %', new.council_district, old.council_district;
+        --
+        -- get engineering area
+        --
+        new.engineering_area_id = (
+            select
+                area_id
+            from
+                public.engineering_areas
+            where
+                st_contains(geometry, new.position)
+            limit 1);
+        raise debug 'engineering_area_id: % compared to previous: %', new.engineering_area_id, old.engineering_area_id;
+        else
+            raise debug 'setting location id and council district to null';
+            -- nullify position column
+            new.position = null;
+            -- reset location id
+            new.location_id = null;
+            -- use city id to determine full purpose jurisdiction
+            new.in_austin_full_purpose = coalesce(new.rpt_city_id = 22, false);
+            raise debug 'setting in_austin_full_purpose based on city id: %', new.in_austin_full_purpose;
+            -- reset council district
+            new.council_district = null;
+            -- reset engineering area
+            new.engineering_area_id = null;
+    end if;
+    return new;
+end;
+$function$