[CT-1751] Config to optionally skip population of relation cache

[jtcohen6]

Users should have the ability to turn off relation cache population, if they really need to. It should still be "on" by default.

$ dbt --no-populate-cache [run|test|...]
$ DBT_POPULATE_CACHE=0|1 dbt [run|test|...]

# profiles.yml (UserConfig)
config:
  populate_cache: true|false

This is different from entirely disabling or skipping over the cache — we're just skipping the population of the cache on startup. When dbt needs to run caching queries, I think it should still report "cache miss," and then cache the result of the metadata query, if it needs to be used again in the same invocation.

Where

dbt-core/core/dbt/task/runnable.py

Lines 407 to 421 in 5453840

	def populate_adapter_cache(self, adapter, required_schemas: Set[BaseRelation] = None):
	start_populate_cache = time.perf_counter()
	if flags.CACHE_SELECTED_ONLY is True:
	adapter.set_relations_cache(self.manifest, required_schemas=required_schemas)
	else:
	adapter.set_relations_cache(self.manifest)
	cache_populate_time = time.perf_counter() - start_populate_cache
	if dbt.tracking.active_user is not None:
	dbt.tracking.track_runnable_timing(
	{"adapter_cache_construction_elapsed": cache_populate_time}
	)
	def before_run(self, adapter, selected_uids: AbstractSet[str]):
	with adapter.connection_named("master"):
	self.populate_adapter_cache(adapter)

Who is it for?

• Users in large projects, or of data warehouses that are quite slow at running metadata queries.
• Users running into inexplicable issues with specific relations not showing up in the relation cache ([CT-1333] [Bug] adapter.get_relation() does not find relation and no way to disable relation cache in dbt run --select #6050)
• Interactive compile & preview ([CT-1584] [Feature] New top level commands: interactive compile #6358, [CT-1585] [Feature] New top level commands: interactive preview #6359), which need to be blazing-fast

YMMV

End users will need to experiment with the approach that's most efficient for them, between:

• full cache enabled
• "cache selected only" (docs)
• --no-populate-cache

I expect mileage may vary between dev, CI, and prod environments.

Questions

Would this break behavior around --defer, which expects to use the relation cache to determine if model X already exists in the dev schema, or should have its reference rewritten to use the schema defined in the other manifest?

dbt-core/core/dbt/contracts/graph/manifest.py

Line 1018 in 5453840

not adapter.get_relation(current.database, current.schema, current.identifier)

Imagining a future where interactive compile/preview want to be both very fast, and able to correctly leverage --defer: We should also think more about making the adapter cache pluggable, as something that can live & persist outside of a single dbt-core invocation. It would be the responsibility of that other application wrapping dbt-core to handle cache invalidation (fun!).

[AnotherGuitar]

We're using the dbt-databricks adaptor and would be very interested in seeing this feature get developed because compile times are taking a while for us. Thanks for writing up this issue!