"dbt run" triggers identical information schema query for each model

[clausherther]

When running dbt with multiple models, dbt issues the same information_schema query for every model in the run, asking for a list of tables in the production schema like so:

select
  TABLE_NAME as name,
  TABLE_TYPE as type
from INFORMATION_SCHEMA.TABLES
where TABLE_SCHEMA in ('<DW Schema>')

Depending on the database, this can up to 1-2 seconds per model and can add a lot of overhead to a run.

Since these queries are called before any transformations are actually done, the queries should all return the same list of tables, so ideally this could be done just once and then cached.
Happy to add more context if needed.

cc: @drewbanin @jthandy

[drewbanin]

Thanks for the writeup @clausherther! We use this query to determine if each table exists, and if so, whether it's a view or a table.

I think you're totally right -- there's minimal risk involved with just running this query once at the beginning of the run. On something like Redshift this doesn't add too much time to the run, but it's definitely painful on Snowflake.

[drewbanin]

see also: #911

I retract my claim that

there's minimal risk involved with just running this query once at the beginning of the run

as it's a little bit complicated, but definitely doable. Check out the issue for more info. Snowflake is pretty easy, whereas supporting this correctly on postgres/redshift is a little bit more involved.

[drewbanin]

This was fixed in #1025