Should we create a terraform package for our Snowflake ELT setup?

[ian-r-rose]

Since #100, we've extracted our snowflake configuration into a few Terraform modules. This makes dev/prod splitting significantly simpler. But it also makes it possible to publish them as separate modules that can be consumed by other repositories (e.g., MDSA projects).

I'm a bit loathe to publish new packages for DIF projects. But there are some advantages to doing so:

1. Allowing MDSA projects to receive updates to the setup.
2. Reducing the lines-of-code needed for an MDSA project.

[ian-r-rose]

An update here: an approach I'm experimenting with is to directly point to the elt snowflake module using GitHub as a module source (this was a major reason for wanting to open source this repo). An example of this is https://github.com/cagov/caldata-mdsa-calhr-ecos/blob/d7f1a95d095dfb103043104cf24311f6f95b77c7/terraform/environments/dev/main.tf#L63-L72

It seems to be working pretty well, and drastically cuts down on the amount of code needed to replicate our environment in a project environment.

There are some downsides to doing this:

1. It tightly couples downstream projects to this terraform configuration. That means that it effectively a published module, and we need to be more careful about making breaking changes to the upstream configuration (e.g., get comfortable with the moved block)
2. It is somewhat less configurable than creating a dedicated module per project. Any needed changes would first need to be made upstream.
3. Any security reviews, etc of a downstream project would likely need to come here to view the configuration.

[ian-r-rose]

I think with the above approach I'm okay with back-burnering this question for a while