cityofaustin · johnclary · Oct 3, 2024 · Sep 13, 2024 · Sep 13, 2024 · Sep 13, 2024
diff --git a/README.md b/README.md
@@ -38,13 +38,13 @@ RR_DATABASE=""
 3. Start the DB
 
 ```shell
-$ vision-zero replicate-db
+./vision-zero replicate-db
 ```
 
 This command will:
 
 - Download a snapshot of the production database
-- Store the file in `./atd-vzd/snapshots/visionzero-{date}-{with|without}-change-log.sql
+- Store the file in `./database/snapshots/visionzero-{date}-{with|without}-change-log.sql
 - Drop local `atd_vz_data` database
 - Create and repopulate the database from the snapshot
 
@@ -53,7 +53,7 @@ Note: the `-f / --filename` flag can be optionally used to point to a specific d
 4. Start the Vision Zero Editor
 
 ```shell
-$ vision-zero vze-up
+vision-zero vze-up
 ```
 
 See [here](docs/local_dev.md) for in-depth documentation about the `vision-zero` helper.
@@ -64,8 +64,8 @@ See [here](docs/local_dev.md) for in-depth documentation about the `vision-zero`
 - [Vision Zero Editor (VZE)](./editor/README.md) - web application which enables City staff to browse and edit crash data
 - [Vision Zero Viewer (VZV)](./viewer/README.md) - public dashboard which provides key metrics and insights about
 - [atd-cr3-api](./atd-cr3-api/README.md) - our API that securely downloads a private file from S3
-- [atd-etl](./etl/README.md) - integration scripts for bringing data in the Vision Zero Database
-- [atd-toolbox](./toolbox/README.md) - collection of utilities and scripts designed for ad-hoc maintenance tasks
+- [etl](./etl/README.md) - integration scripts for bringing data in the Vision Zero Database
+- [toolbox](./toolbox/README.md) - collection of utilities and scripts designed for ad-hoc maintenance tasks
 
 ## License
 

diff --git a/database/Dockerfile b/database/Dockerfile
@@ -1,3 +1,3 @@
 FROM frankinaustin/postgis-multiarch:14-3.3
 RUN apt-get update
-RUN apt-get install -y aptitude magic-wormhole vim
+RUN apt-get install -y aptitude vim
diff --git a/database/README.md b/database/README.md
@@ -2,7 +2,7 @@
 
 The Vision Zero Database (VZD) is a Postgresql database that serves as the central repository of Austin's traffic crash data. The database is fronted with a GraphQL API, powered by [Hasura](https://github.com/hasura/graphql-engine), which is also used to manage schema migrations.
 
-The design supports an editing environment which enables Vision Zero program staff to edit and enrich crash data, while also allowing record updates to flow into the database from upstream sources, such as the TxDOT Crash Information System (CRIS).
+The design supports an editing environment which enables Vision Zero program staff to edit and enrich crash data, while also allowing record updates to flow into the database from upstream sources, such as the TxDOT Crash Records Information System (CRIS).
 
 ![vision zero data flow](../docs/images/data_flow.png)
 
@@ -19,7 +19,7 @@ The design supports an editing environment which enables Vision Zero program sta
     - [Audit fields](#audit-fields)
     - [Change logs](#change-logs)
   - [Austin Fire Department (AFD) and Travis County Emergency Medical Services (EMS) (todo)](#austin-fire-department-afd-and-travis-county-emergency-medical-services-ems-todo)
-  - [Geospatial layers (todo)](#geospatial-layers-todo)
+  - [Geospatial layers](#geospatial-layers)
 - [Common maintenance tasks](#common-maintenance-tasks)
   - [Add a new CRIS-managed column to `crashes`, `units`, or `people`](#add-a-new-cris-managed-column-to-crashes-units-or-people)
   - [Add a custom column to `crashes`, `units`, or `people`](#add-a-custom-column-to-crashes-units-or-people)
@@ -75,7 +75,41 @@ The process for updating `units` and `people` behaves in the same manner as `cra
 
 #### CRIS Extract configuration and accounts
 
-TODO. See https://app.gitbook.com/o/-LzDQOVGhTudbKRDGpUA/s/-M4Ul-hSBiM-3KkOynqS/vision-zero-crash-database/visionzero-sftp-etl#getting-extracts-manually
+The team maintains multiple CRIS login accounts to manage the delivery of CRIS data to our AWS S3 ingest bucket. See the [CRIS import ETL documentation](../etl/cris_import/README.md) for more information on how CRIS data is delivered to S3 and imported into our database.
+
+The credentials for our CRIS logins are in the password store, including a note in the title indicating each login's purpose:
+
+- **Production extract account**: this is the account which is configured for daily delivery of extracts to S3 production.
+- **Dev/testing extract account**: this account should be used for requesting ad-hoc CRIS extracts for delivery via S3 or manual download.
+- **Query & analyze account**: this account can be used for the CRIS query interface, that enables querying and access to individual crash records.
+
+Additional information about CRIS access can be found on the [TxDOT website](https://www.txdot.gov/data-maps/crash-reports-records/crash-data-analysis-statistics.html).
+
+Follow these steps to configure a new extract delivery:
+
+1. Login to CRIS using the appropriate account (see above): https://cris.dot.state.tx.us/
+
+2. From the **My Extract Requests** page, click **Add**, and follow the extract request wizard
+
+- **Extract Type**: Standard
+- **Extract Format**: CSV
+- **Include CR-3 Crash Report files in Extract**: Yes (checked)
+
+![CRIS extract config - page 1](../docs/images/extract_config_1.png)
+
+- **Include Crash Reports From**: Specific Counties: `Hays`, `Travis`, and `Williamson`
+
+- **Include Crash Reports From**: Process Date range
+  - If you are backfilling, include a day before your target day as a buffer.
+  - If you request a process date of today, the extract will not deliver until the next day.
+  - To set up a recurring request, add a range of dates that ends in the future.
+
+Any part of the range that falls in the past will be delivered in single zip that is separate from the zips that will deliver in the future. The includes - all records with process dates available including today.
+
+Any part of the range that is in the future will create daily zips that include each day available going forward. For example, on 4/19/2024, you make a request for Process Begin Date = 01/01/2024 and Process End Date = 12/31/2024 The would receive two zips: One containing all records with process date from 01/01/2024 to 04/18/2024, and one containing all records with process date from 04/19/2024 to 04/19/2024. Going forward, you will receive one zip per day for each process date that passes
+
+- **Extract password**: the password called `EXTRACT_PASSWORD` from Vision Zero CRIS Import 1Password item
+- **Delivery**: How you want to receive it. Typically you would use the pre-configured AWS option, specifiyng the `dev`, `staging`, or `prod` inbox subdirectory. See the CRIS import ETL readme for more details.
 
 #### CRIS data processing
 
@@ -121,7 +155,7 @@ For example, consider the `lookups.injry_sev` table, which includes a custom val
 | 99  | KILLED (NON-ATD)         | vz     |
 ```
 
-The original migration for this table is [here](https://github.com/cityofaustin/atd-vz-data/blob/e56e3c6bc654a21f667142ce53232bad44cff7e5/database/migrations/default/1715960018005_lookup_table_seeds/up.sql#L11054-L11056).
+The original migration for this table is [here](https://github.com/cityofaustin/vision-zero/blob/e56e3c6bc654a21f667142ce53232bad44cff7e5/atd-vzd/migrations/default/1715960018005_lookup_table_seeds/up.sql#L11054-L11056).
 
 Because the table has a custom value, it is configured with a check constraint ([PostgreSQL docs](https://www.postgresql.org/docs/current/ddl-constraints.html#DDL-CONSTRAINTS-CHECK-CONSTRAINTS)) to ensure that future updates to this lookup table do not result in an ID collision:
 
@@ -199,19 +233,23 @@ The view `crashes_change_log_view` provides a unioned view of the unified table
 
 ### Austin Fire Department (AFD) and Travis County Emergency Medical Services (EMS) (todo)
 
-### Geospatial layers (todo)
+### Geospatial layers
+
+We have a number of tables which function as geospatial layers which are referenced by crashes and various other records. At the Vision Zero team's request, our team is actively working to expand the number of layers available in the database as well as add new attribute columns to crash records which will be populated based on their intersection with these layers.
 
-- Council districts
-- Jurisdiction
-- Area Engineer areas
-- Non-COA roadways
-- Location polygons
+| Table                 | Geometry type | description                                                                                               | owner/source                                                         |
+| --------------------- | ------------- | --------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- |
+| `council_districts`   | polygon       | City of Austin council districts                                                                          | ArcGIS Online authoritative layer owned by CTM GIS                   |
+| `atd_jurisdictions`   | polygon       | City of Austin jurisdictions                                                                              | ArcGIS Online authoritative layer owned by CTM GIS                   |
+| `engineering_areas`   | polygon       | TPW traffic engineering areas                                                                             | ArcGIS Online authoritative layer owned by DTS GIS                   |
+| `non_coa_roadways`    | polygon       | Polygon layer covering roadways which are not maintained by the City of Austin                            | ArcGIS Online authoritative layer maintained by Vision Zero GIS team |
+| `atd_txdot_locations` | polygon       | Aka, "location polygons", these shapes are used to group crashes based on an intersection or road segment | ArcGIS Online authoritative layer maintained by Vision Zero GIS team |
 
 ## Common maintenance tasks
 
 ### Add a new CRIS-managed column to `crashes`, `units`, or `people`
 
-Follow these steps to add a new column to the database that will be sourced from CRIS. See [PR #1546](https://github.com/cityofaustin/atd-vz-data/pull/1546) as an example.
+Follow these steps to add a new column to the database that will be sourced from CRIS. See [PR #1546](https://github.com/cityofaustin/vision-zero/pull/1546) as an example.
 
 1. Remember that all database operations should be deployed through migrations. See the [development and deployment](#development-and-deployment) docs.
 2. Add the new column to all three tables of the given record type. For example, if this is a crash-level column, add the column to the `crashes_cris`, `crashes_edits`, and `crashes` tables.
@@ -229,7 +267,7 @@ values ('drvr_lic_type_id', 'people', true);
 
 ```shell
 # ./database
-$ hasura metadata apply
+hasura metadata apply
 ```
 
 7. You are now ready to test your new column using the CRIS import ETL. If you need to backfill this new column for old records, you will need to manually request the necessary CRIS extract zip files so that they can be processed by an ad-hoc run of the CRIS import ETL.

diff --git a/database/metadata/databases/default/tables/public_view_crash_narratives_ocr_todo.yaml b/database/metadata/databases/default/tables/public_view_crash_narratives_ocr_todo.yaml
@@ -0,0 +1,3 @@
+table:
+  name: view_crash_narratives_ocr_todo
+  schema: public
diff --git a/database/metadata/databases/default/tables/tables.yaml b/database/metadata/databases/default/tables/tables.yaml
@@ -86,3 +86,4 @@
 - "!include public_units.yaml"
 - "!include public_units_cris.yaml"
 - "!include public_units_edits.yaml"
+- "!include public_view_crash_narratives_ocr_todo.yaml"
diff --git a/database/migrations/default/1727375054583_narrative_ocr/down.sql b/database/migrations/default/1727375054583_narrative_ocr/down.sql
@@ -0,0 +1,7 @@
+drop view if exists view_crash_narratives_ocr_todo;
+
+alter table crashes_edits
+    drop column investigator_narrative_ocr_processed_at;
+
+alter table crashes
+    drop column investigator_narrative_ocr_processed_at;
diff --git a/database/migrations/default/1727375054583_narrative_ocr/up.sql b/database/migrations/default/1727375054583_narrative_ocr/up.sql
@@ -0,0 +1,39 @@
+alter table crashes_edits
+add column investigator_narrative_ocr_processed_at timestamp with time zone;
+
+alter table crashes
+add column investigator_narrative_ocr_processed_at timestamp with time zone;
+
+comment on column crashes_edits.investigator_narrative_ocr_processed_at is 'The most recent
+timestamp at which the OCR process attempted to extract the investigator narrative. If null, 
+indicates that the OCR narrative extract has never been attempted. This value should be set
+via ETL process.';
+
+comment on column crashes.investigator_narrative_ocr_processed_at is 'The most recent
+timestamp at which the OCR process attempted to extract the investigator narrative. If null, 
+indicates that the OCR narrative extract has never been attempted. This value should be set
+via ETL process on the crashes_edits table.';
+
+create or replace view view_crash_narratives_ocr_todo as (
+    select
+        id,
+        cris_crash_id
+    from
+        crashes
+    where
+        cr3_stored_fl = TRUE
+        and investigator_narrative is NULL
+        and (
+            investigator_narrative_ocr_processed_at is NULL
+            or cr3_processed_at >= investigator_narrative_ocr_processed_at
+        )
+        -- this issue started in Sep 2024
+        -- we do not OCR very old crashes
+        and updated_at > '2024-09-01'
+    order by
+        cr3_processed_at asc,
+        id asc
+);
+
+comment on view view_crash_narratives_ocr_todo is 'View which lists crashes which need to 
+be processed by the OCR narrative extraction ETL'
diff --git a/database/migrations/default/1727451510064_preserve_crash_narrative/down.sql b/database/migrations/default/1727451510064_preserve_crash_narrative/down.sql
@@ -0,0 +1,5 @@
+drop trigger if exists
+crashes_cris_preserve_investigator_narrative_on_update on crashes_cris;
+
+drop function if exists
+public.crashes_cris_set_old_investigator_narrative;
diff --git a/database/migrations/default/1727451510064_preserve_crash_narrative/up.sql b/database/migrations/default/1727451510064_preserve_crash_narrative/up.sql
@@ -0,0 +1,50 @@
+create or replace function public.crashes_cris_set_old_investigator_narrative()
+    returns trigger
+    language plpgsql
+    as $function$
+begin
+    new.investigator_narrative = old.investigator_narrative;
+    return new;
+end;
+$function$;
+
+create or replace trigger crashes_cris_preserve_investigator_narrative_on_update
+before update on public.crashes_cris
+for each row
+when (
+    new.investigator_narrative is null and old.investigator_narrative is not null
+)
+execute procedure public.crashes_cris_set_old_investigator_narrative();
+
+comment on function public.crashes_cris_set_old_investigator_narrative is 'Sets the
+investigator_narrative to its previous value if the updated value is null. This 
+trigger function addresses a known CRIS bug in which updated crash records are 
+missing the invesitgator narrative. It is tracked via DTS issue 
+https://github.com/cityofaustin/atd-data-tech/issues/18971 and CRIS ticket #854366';
+
+--
+-- backfill narratives which have been erased
+-- run this manually to prevent migration timeout
+--
+
+-- update
+--     crashes_cris
+-- set
+--     investigator_narrative = updates_todo.investigator_narrative_old
+-- from (select
+--     record_id as crash_pk,
+--     crashes.investigator_narrative as investigator_narrative_new,
+--     record_json -> 'old' ->> 'investigator_narrative' as investigator_narrative_old
+-- from
+--     change_log_crashes_cris as changes
+--     left join crashes on changes.record_id = crashes.id
+-- where
+--     record_json -> 'old' ->> 'investigator_narrative' is not null
+--     and record_json -> 'new' ->> 'investigator_narrative' is null
+--     and operation_type = 'UPDATE'
+--     and changes.created_at > '2024-09-09'
+--     and changes.created_by = 'cris'
+-- order by
+--     changes.id asc) as updates_todo
+-- where
+--     crashes_cris.id = updates_todo.crash_pk;
diff --git a/docker-compose-docker-volume.yml b/docker-compose-docker-volume.yml
@@ -6,6 +6,7 @@ services:
       - .env
     volumes:
       - visionzero_postgis_pgdata:/var/lib/postgresql/data
+      - ./database/snapshots:/snapshots
     ports:
       - 5432:5432
 

diff --git a/docker-compose.yml b/docker-compose.yml
@@ -1,6 +1,6 @@
 services:
   graphql-engine:
-    image: hasura/graphql-engine:v2.40.2
+    image: hasura/graphql-engine:v2.43.0
     volumes:
       - ./viewer/graphql-engine-metadata:/metadata
     container_name: visionzero-graphql-engine
@@ -10,17 +10,6 @@ services:
       HASURA_GRAPHQL_DATABASE_URL: postgres://visionzero:visionzero@postgis:5432/atd_vz_data
       HASURA_GRAPHQL_ENABLE_CONSOLE: "true"
       HASURA_GRAPHQL_CONSOLE_ASSETS_DIR: /srv/console-assets
-  db-tools:
-    container_name: visionzero_download_db_data
-    logging:
-      driver: none
-    build: toolbox/download-db-data
-    command: tail -f /dev/null
-    hostname: db-tools
-    env_file:
-      - .env
-    volumes:
-      - ./database/snapshots:/snapshots
   vze:
     tty: true
     container_name: visionzero-vze

diff --git a/docs/images/extract_config_1.png b/docs/images/extract_config_1.png
diff --git a/docs/local_dev.md b/docs/local_dev.md
@@ -61,7 +61,7 @@ Stop the stack
 #### `vision-zero replicate-db`
 
 - Download a snapshot of the production database
-- Store the file in `./atd-vzd/snapshots/visionzero-{date}-{with|without}-change-log.sql`
+- Store the file in `./database/snapshots/visionzero-{date}-{with|without}-change-log.sql`
 - Drop local `atd_vz_data` database
 - Create and repopulate the database from the snapshot
 
@@ -74,7 +74,7 @@ one copy of the data per-day, both in the with and without change log data.
 #### `vision-zero dump-local-db`
 
 - pg_dump the current local database
-- Stores the file in `./atd-vzd/dumps/visionzero-{date}-{time}.sql
+- Stores the file in `./database/dumps/visionzero-{date}-{time}.sql
 
 #### `vision-zero remove-snapshots`