Snowflake Destination V2 still trying to utilize V1 raw tables

[Jordonkopp]

Connector Name

destination-snowflake

Connector Version

3.4.10

What step the error happened?

During the sync

Relevant information

Suggested from here to create a new ticket.

When upgrading to Snowflake Destination > 3.x.x it is still trying to use the V1 raw tables over the V2 raw tables.

This error is "hidden" on upgrading from <3.x.x because those V1 raw tables still exists and most likely will lead to bad data since those V1 tables are no longer updated.

We came across this while testing other things and needed to create new connections, which will never complete due to this issue.

This can easily be reproduced by just creating a new connection pointing to a new schema/new prefix where the V1 raw tables would not exist for the table you are syncing.

We are stuck to Snowflake Destination 2.1.7 until this is resolved

c026278b_cd82_453e_a895_00935178025d_logs_2391_txt.txt

Relevant log output

2023-11-14 01:07:33 �[32mINFO�[m i.a.w.g.DefaultNormalizationWorker(run):109 - Normalization summary: io.airbyte.config.NormalizationSummary@2a41a8f[startTime=1699924027939,endTime=1699924053792,failures=[io.airbyte.config.FailureReason@4f70399[failureOrigin=normalization,failureType=system_error,internalMessage=002003 (42S02): SQL compilation error: Object 'INFOSCOUT_AIRBYTE_STG.JK_TEST._AIRBYTE_RAW_INTERNATIONAL_LANGUAGE' does not exist or not authorized.,externalMessage=Normalization failed during the dbt run. This may indicate a problem with the data itself.,metadata=io.airbyte.config.Metadata@4a9f9af[additionalProperties={attemptNumber=0, jobId=2391, from_trace_message=true}],stacktrace=AirbyteDbtError: 
1 of 3 ERROR creating view model _AIRBYTE_JK_TEST.INTERNATIONAL_LANGUAGE_STG............................................ [�[31mERROR�[0m in 0.78s]
�[33mDatabase Error in model INTERNATIONAL_LANGUAGE_STG (models/generated/airbyte_views/JK_TEST/INTERNATIONAL_LANGUAGE_STG.sql)�[0m
  002003 (42S02): SQL compilation error:
  Object 'INFOSCOUT_AIRBYTE_STG.JK_TEST._AIRBYTE_RAW_INTERNATIONAL_LANGUAGE' does not exist or not authorized.
  compiled SQL at ../build/run/airbyte_utils/models/generated/airbyte_views/JK_TEST/INTERNATIONAL_LANGUAGE_STG.sql
1 of 3 ERROR creating view model _AIRBYTE_JK_TEST.INTERNATIONAL_LANGUAGE_STG............................................ [�[31mERROR�[0m in 0.78s]
�[33mDatabase Error in model INTERNATIONAL_LANGUAGE_STG (models/generated/airbyte_views/JK_TEST/INTERNATIONAL_LANGUAGE_STG.sql)�[0m
  002003 (42S02): SQL compilation error:
  Object 'INFOSCOUT_AIRBYTE_STG.JK_TEST._AIRBYTE_RAW_INTERNATIONAL_LANGUAGE' does not exist or not authorized.
  compiled SQL at ../build/run/airbyte_utils/models/generated/airbyte_views/JK_TEST/INTERNATIONAL_LANGUAGE_STG.sql,retryable=<null>,timestamp=1699924052715,additionalProperties={}], io.airbyte.config.FailureReason@2e229b15[failureOrigin=normalization,failureType=system_error,internalMessage=002003 (42S02): SQL compilation error: Object 'INFOSCOUT_AIRBYTE_STG.JK_TEST._AIRBYTE_RAW_INTERNATIONAL_LANGUAGE' does not exist or not authorized.,externalMessage=Normalization failed during the dbt run. This may indicate a problem with the data itself.,metadata=io.airbyte.config.Metadata@c709c3c[additionalProperties={attemptNumber=0, jobId=2391, from_trace_message=true}],stacktrace=AirbyteDbtError: 
1 of 3 ERROR creating view model _AIRBYTE_JK_TEST.INTERNATIONAL_LANGUAGE_STG............................................ [�[31mERROR�[0m in 0.78s]
�[33mDatabase Error in model INTERNATIONAL_LANGUAGE_STG (models/generated/airbyte_views/JK_TEST/INTERNATIONAL_LANGUAGE_STG.sql)�[0m
  002003 (42S02): SQL compilation error:
  Object 'INFOSCOUT_AIRBYTE_STG.JK_TEST._AIRBYTE_RAW_INTERNATIONAL_LANGUAGE' does not exist or not authorized.
  compiled SQL at ../build/run/airbyte_utils/models/generated/airbyte_views/JK_TEST/INTERNATIONAL_LANGUAGE_STG.sql
1 of 3 ERROR creating view model _AIRBYTE_JK_TEST.INTERNATIONAL_LANGUAGE_STG............................................ [�[31mERROR�[0m in 0.78s]
�[33mDatabase Error in model INTERNATIONAL_LANGUAGE_STG (models/generated/airbyte_views/JK_TEST/INTERNATIONAL_LANGUAGE_STG.sql)�[0m
  002003 (42S02): SQL compilation error:
  Object 'INFOSCOUT_AIRBYTE_STG.JK_TEST._AIRBYTE_RAW_INTERNATIONAL_LANGUAGE' does not exist or not authorized.
  compiled SQL at ../build/run/airbyte_utils/models/generated/airbyte_views/JK_TEST/INTERNATIONAL_LANGUAGE_STG.sql,retryable=<null>,timestamp=1699924052715,additionalProperties={}]],additionalProperties={}]

Contribute

• Yes, I want to contribute

[Jordonkopp]

@evantahler bump on this per conversation here

[friendofasquid]

NB: don't be like me and try to create the missing table. This will end up overwriting your synced data.

[evantahler]

We will be investigating this over in https://github.com/airbytehq/airbyte-internal-issues/issues/2752. Thanks for reporting this!