[ADAP-464] Materialized Views - Non-incremental support

[Fleid]

As commented in the core MV thread, BigQuery is adding support for non-incremental MVs.

From our side, it means that we have an additional setting to support for MVs in BQ: allow_non_incremental_definition: true/false. This setting is covered by on_configuration_change via DROP/CREATE since it can't be changed via ALTER.

When set, it requires max_staleness to be set. I'd rather leave BQ errors out here for now - this is a preview feature and I'm not sure this won't change before GA. We should not check this.

---

BigQuery

BigQuery Materialized Views

• Configurations:
  • enable_refresh : BOOLEAN, optional (database level default true) (see lifecycle for effect)
  • refresh_interval_minutes : integer, optional (database level default at 30), only valid for enable_refresh true or default
  • max_staleness : time before bypassing the MV and running the statement instead (in preview but I like it!)
  • allow_non_incremental_definition : BOOLEAN, optional (database default false)
  • From models: Partition / Cluster, TTL, Labels
• Lifecycle : default except Refresh is bypassed if auto_refresh
  • on_configuration_change.apply : enable_refresh/refresh_interval_minutes/max_staleness via ALTER, rest via DROP/CREATE

[Fleid]

@mikealfare FYI

[bnaul]

@Fleid @mikealfare this has been out there for a while now, I think it's basically impossible at this point that they would drastically change the functionality, as they have written it up and pulled it into official client libraries. Happy to try my hand at a PR for adding this if it would be welcome! Changing to an "ask for forgiveness instead of permission" approach #1011

[LuigiCerone]

Hello, any update on this? :)

[kangshung]

Bump @mikealfare do you have this feature in a roadmap?

It's already GA. link

[yu-iskw]

It seems that the pull requests #1011 and #1255 aren't actively going. So, let me try to take over to support the feature.

#1417