Epic - Open Data Publishing 2023

[tiffanychu90]

After receiving a research request, use this template to plan and track your work. Be sure to also add the appropriate project-level label to this issue (eg gtfs-rt, DLA).

Epic Information - Open Data Publishing 2023

Summary

• Easily track the PRs related to monthly updates to open data portal
• Going forward, no need to open an issue each month, just link PR here
• Reference: Monthly dates downloaded for open data
• Publish 6 geospatial datasets each month:
  • ca_hq_transit_areas (HQTA polygons)
  • ca_hq_transit_stops (HQTA points)
  • ca_transit_routes (initially Traffic Ops request)
  • ca_transit_stops (initially Traffic Ops request)
  • speeds_by_stop_segments
  • speeds_route_time_of_day
• Steps to produce open data portal:
  • add current month's date in shared_utils/rt_dates
  • download GTFS scheduled trips, stops, shapes, stop_times, and vehicle_locations tables for a single day: cd gtfs_funnel/ && make funnel_gtfs_data
  • HQ areas/stops datasets: cd high_quality_transit_areas/ && make hqta_data (interspersed with notebook checks)
  • GTFS stops/routes geospatial datasets: cd open_data/ && make create_gtfs_schedule_geospatial_open_data
  • update metadata XML and publish to enterprise geodatabaes: cd open_data/ && make compile_open_data_portal (this step is more manual)

Issues

2023 open data (stable phase)

• Breaking change: using v2 warehouse: Research Request - Move all open data portal outputs to v2 #607
• Jan 2023: Hqta v2 warehouse #614
• Feb 2023: Feb open data #654
• Mar 2023: Mar open data #686
• Apr 2023: Apr open data #721
• May 2023: May open data #756
• Jun 2023: Jun open data #786
• Jul 2023: Jul open data #808 (first month for 2 speed datasets to be included)
• Aug 2023: Aug open data #840, Find muni #855
• Sep 2023: Sep open data #878
• Oct 2023: Oct open data #929, Segment refactoring #932
• Nov 2023: Nov open data #952 (first month for interpolated methodology for speeds)
• Dec 2023: Dec open data #967

2022 open data (development phase)

• Dec 2022: Dec open data #580
• Nov 2022: Nov open data #553
• Oct 2022: Research Request - Oct open data (HQTA, Traffic Ops) #480
• Sep 2022: Research Task - Sep open data portal datasets #449

Wrangling data for publishing

• 1. HQTA v1
  Quick census hqta #11
  Hqta format #12
  Hqta revisions #22
  Hqta updates #39
  Feb hqta update #130
  May hqta #248
• 2. first pass at publishing HQTA and Traffic Ops
  Research: routes and stops shapefiles for public-facing portal #160
  User Story: HQTA Data Available on CA Open Data Portal #174
  Research Task - Add metadata for routes/stops shapefile #283
  rerun hqta for june, add explanatory column #297
• 3. HQTA rewrites
  Epic - Daskify HQTA #321
• 4. Traffic Ops rewrite
  Research Task - Move traffic_ops open data over to use gtfs_utils #352

Enterprise Geodatabase

• 1. getting initial metadata set up
  Research Task - Overwrite ESRI metadata XML  #281
  Research Task - Add metadata for routes/stops shapefile #283
  Research Task - Fix column names for traffic ops routes/stops open data #296
  Research Task - Add horizontal accuracy to open data #304
• 2. from zipped file geodatabase to enterprise geodatabase
  Research Task - Put August HQTA dataset into ESRI enterprise geodatabase #399
• 3. moving from ArcGIS desktop to ArcGIS Pro
  Research Task - Rewrite metadata update steps using ArcGIS Pro #429
  Bug: ArcGIS Pro enterprise gdb metadata...showing default template values #508
• 4. changes to Caltrans required metadata items
  Update metadata #802, More metadata #810, Research Task - ArcPro template for updating the definition associated with each field #813
• 5. investigate any options to make metadata updates easier
  Research Task - See if ArcPro can write geoparquets to file geodatabase's feature classes #953

[tiffanychu90]

Close epic for 2023 PRs related to open data publishing. Use new epic for 2024.