[ADAP-503] [Regression] Persisted Metadata does not update on BigQuery tables

[patricker]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

After upgrading to dbt 1.5.0 column metadata does not update in BigQuery and the model build shows as failed.

Expected Behavior

metadata writes to the tables without issue in BigQuery.

Steps To Reproduce

1. with dbt 1.5.0, writing to BigQuery
2. Update your dbt_project:
   +persist_docs:
   relation: true
   columns: true
3. Run a model with column level documentation
4. Error received after model is fully built, during the docs persist phase.

Relevant log output

Error: google.api_core.exceptions.BadRequest: 400 PATCH https://bigquery.googleapis.com/bigquery/v2/projects/{table_path}?prettyPrint=false: Policy tags can only be attached to a leaf field.

Environment

- OS:debian
- Python:3.9.10
- dbt:1.5.0

Which database adapter are you using with dbt?

bigquery

Additional Context

I rolled back to 1.4.6 and the issue went away.

[dbeatty10]

Thanks for reporting this @patricker and @steve-marmalade.

I was able to reproduce this when using policy_tags and persist_docs.

[dbeatty10]

Simple reprex

dbt_project.yml

name: "my_dbt_project"
version: "1.0.0"
config-version: 2
profile: "sandcastle-bigquery"

models:
  my_dbt_project:
    +persist_docs:
      relation: true
      columns: true

models/_models.yml

# version isn't required for dbt-core 1.5, but it is for 1.4
version: 2

models:
  - name: my_model
    columns:
      - name: id
        description: Primary key
        policy_tags:
          # Example pattern:
          # - 'projects/<gcp-project>/locations/<location>/taxonomies/<organization>/policyTags/<tag>'
          - projects/dbt-test-env/locations/us/taxonomies/5785568062805976401/policyTags/135489647357012267

models/my_model.sql

{{ config(materialized='table') }}

select 1 as id

[dgreen161]

I've written a fix for this issue as above. It looks like this bug was introduced during this MR which attempts to remove policy tags from all columns and structs on the model.

I wasn't able to reproduce it using the method that you described @dbeatty10 but I could produce it using the below setup:

dbt_project.yml

name: "my_dbt_project"
version: "1.0.0"
config-version: 2
profile: "sandcastle-bigquery"

models:
  my_dbt_project:
    +persist_docs:
      relation: true
      columns: true

my_model.sql

select
    struct(
        1 AS id
    ) as first_struct

_models.yml

version: 2

models:
  - name: my_model
    description: "A starter dbt model"
    columns:
      - name: first_struct
      - name: first_struct.id

It fails when trying to remove policy tags from the struct as defined in the yml which was introduced in the MR 542. You would want structs in the yml so you can apply descriptions to them in the model in BigQuery.

[dbeatty10]

Thanks for truly reproducing this @dgreen161 🙏

Once I looked more closely at the error I "reproduced", it was actually a 403 rather than a 400 😅

[dgreen161]

Hi @nssalian @dbeatty10 Do you have a timeline for when the backport will be accessible through pip and dbt Cloud? It looks like 1.5.0 was last released 3 weeks ago. I can see it's in the beta release but that still leaves 1.5 with the issue. Thanks

[dbeatty10]

@dgreen161 Thank you for checking-in!

We just released the backport yesterday -- wanna give it shot?

[dgreen161]

@dbeatty10 Thanks for the update, I've just run the new version (v1.5.1) locally and the issue has now gone