GTFS Schedule optimizations: require partition filters

[atvaccaro]

Description

NOTE: must wait until #2103 is merged this is merged

Closes #2134

This is non-trivial to test for correctness because of a lack of true primary keys in the final dimension tables, but I've verified re-running this does not produce duplicate rows. The underlying macro first filters the possible GTFS data by day, but still inner joins against feed versions.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation
• agencies.yml

How has this been tested?

Testing as best I can, though we may still have some queries that fail. We can monitor the query access logs for failures.

-m +mart.gtfs_schedule_latest run/test passes as much as we would expect. dim_translations fails, we may just want to disable it?

Screenshots (optional)

[lauriemerrell]

I am still not sold on the need for this. I think that we can afford to run all of history for most schedule files with no problems, and I am particularly unsure that 120 days is enough for schedule. Like I think that when we do a full refresh we will basically always end up overriding to get all of history since 2021-04-16.

Can you add more documentation in the ticket or here about the rationale for doing this, and how you picked the defaults? Personally I think that having a mechanism for different lookback between prod and dev is actually more important than changing prod.

[lauriemerrell]

It feels a bit weird to me to have the materialization in a macro. I guess it is ok? But I am a bit 🤔

[atvaccaro]

Yeah it's a bit weird but ideally we just force anything using this macro to be incremental, since the macro itself only works in an incremental model.

Multiple calls to config are explicitly supported, luckily dbt-labs/dbt-core#1150

[atvaccaro]

I am still not sold on the need for this. I think that we can afford to run all of history for most schedule files with no problems, and I am particularly unsure that 120 days is enough for schedule. Like I think that when we do a full refresh we will basically always end up overriding to get all of history since 2021-04-16.

Yes that's fair, I'll change this at minimum.

Can you add more documentation in the ticket or here about the rationale for doing this, and how you picked the defaults? Personally I think that having a mechanism for different lookback between prod and dev is actually more important than changing prod.

The immediate cause of this work was Metabase fully scanning stop_times for purposes of auto-complete; even though it queries individual columns, any query that hits an external table must read the full file, and Metabase was doing no filtering on dt (which we shouldn't expect it to do). More broadly, I want us to start protecting our external tables from expensive queries that we may not realize are occurring without looking at audit logs. Even if we read in all of Schedule history (i.e. set the lookback to 2022-04-16), we'd still be protecting ourselves against expensive reads on stop_times, etc. via Metabase/Jupyter/whatever, human or computer. The fact we then have to filter in dbt is more a consequence than a goal; if you would prefer, I can split this PR into just requiring filters with a dummy where in dbt, and either skip incremental materializations or do them in a separate PR.

[SorenSpicknall]

NOTE: must wait until #2138 is merged

Is this supposed to be self-referencing, or is there another PR that should be referenced as a dependency here?

[atvaccaro]

NOTE: must wait until #2138 is merged

Is this supposed to be self-referencing, or is there another PR that should be referenced as a dependency here?

Ah no good catch, it should've referenced #2103

[atvaccaro]

Here's some example queries re: Slack discussion.

select *
from cal-itp-data-infra-staging.andrew_mart_gtfs.dim_stop_times
where '2023-01-19' between _valid_from and _valid_to; -- 460 gb

select *
from cal-itp-data-infra-staging.andrew_mart_gtfs.dim_stop_times_clustered_by_valids
where '2023-01-19' between _valid_from and _valid_to; -- 460 gb

select *
from cal-itp-data-infra-staging.andrew_mart_gtfs.dim_stop_times_clustered_by_valids
where _valid_from < '2023-01-19' AND '2023-01-19' < _valid_to; -- 13 gb

select *
from cal-itp-data-infra.mart_gtfs.dim_stop_times st
inner join cal-itp-data-infra.mart_gtfs.dim_schedule_feeds sf
  on st.feed_key = sf.key
where '2023-01-19' between sf._valid_from and sf._valid_to; -- 461 gb

select *
from cal-itp-data-infra-staging.andrew_mart_gtfs.dim_stop_times_clustered_by_feed_key st
inner join cal-itp-data-infra.mart_gtfs.dim_schedule_feeds sf
  on st.feed_key = sf.key
where '2023-01-19' between sf._valid_from and sf._valid_to; -- 33 gb

[lauriemerrell]

There's still YAML in _int_gtfs for int_gtfs_schedule__incremental_stop_times and the others (GH won't let me comment inline), can those be deleted?

[lauriemerrell]

see response on comment on the macro

[atvaccaro]

Going to wait post-reports since this changes external tables and has the potential to cause unexpected problems.