Doesn't work with spark 3.4.0

[sid-deshmukh]

Describe the bug

The external tables creation is failing with spark 3.4.0 where as it works with pervious versions

Steps to reproduce

version: 2

sources:

• name: Schema_name
  tables:
  • name: external_table
    description: ""
    external:
    location: s3 path
    using: json

Expected results

External table created in Glue catelog when run from EMR

Actual results

Its fails with below Error:
Encountered an error while running operation: Compilation Error
'NoneType' object is not iterable

in macro stage_external_sources (macros/common/stage_external_sources.sql)
called by

Which database are you using dbt with?

• Spark

[dataders]

@sid-deshmukh can you please share what version of dbt-core dbt-spark and dbt-external-tables you are using? was there no change to these versions besides upgrading the version of Spark itself?

can you share the full error message as well as any relevant section of logs/dbt.log?

[sid-deshmukh]

@dataders There was no dbt-core and dbt-spark version changes. But I have tried with both 1.5.0 and 1.6.0, I get the same error.
I have attached the logs for dbt.log
dbt.log

[dataders]

closing in favor of dbt-labs/dbt-spark#885. @sid-deshmukh I would very much love to see if you can

• get the same error you shared when running dbt run
• reproduce this error in dbt-spark 1.4.0