[CT-2533] [Feature] Support sqlparse 0.4.4

[lukehsiao]

Is this your first time submitting a feature request?

• I have read the expectations for open source contributors
• I have searched the existing issues, and I could not find an existing issue for this feature
• I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature

sqlparse <0.4.4 contains a moderate security vulnerability: GHSA-rrm6-wvj7-cwh2

dbt-core has started to pin to <0.4.4 in #7394 which makes it difficult for packages using dbt to update to the fixed version.

Would it be possible for dbt to support v0.4.4?

Describe alternatives you've considered

The answer might just be "no". In which case we will have to wait until a new version of sqlparse addresses the issue. It does not seem like there is anything upstream sqlparse tracking work to resolve #7396. So, if the answer is no, then I would hope the appropriate effort upstream can be made so we have a path forward eventually.

Who will this benefit?

All users of dbt-core who want to update to a non-vulnerable sqlparse version.

Are you interested in contributing this feature?

No response

Anything else?

No response

[dimoschi]

+1 even a moderate security vulnerability can be a problem for some organizations.

[dbeatty10]

Thanks for reaching out @lukehsiao and @dimoschi 🙏

We might need to wait until a new version of sqlparse addresses the issue. The best next step would be for someone to open a relevant issue upstream with sqlparse so that it has the opportunity to be fixed.

I can take it as an action item to open an issue with sqlparse and provide them with a reproduction case that mimics the failure of the test_singular_tests_ephemeral test in our CI here.

Background for those just tuning in

We introduced a pin of sqlparse<0.4.4 because our functional testing suite was failing for sqlparse==0.4.4, and you can read more here:
#7396 (comment)

Side note: The CI tests actually worked for macOS Windows and it was only failing for Linux (Ubuntu).

[dimoschi]

Gonna try reproduce the issue and look if I can open a PR at sqlparse. Thanks for providing context @dbeatty10

[jtcohen6]

Another user just reported (#7521) seeing the same issue with sqlparse==0.4.4 on Debian GNU/Linux 11 (bullseye)

[dbeatty10]

@MauroLuzzatto provided some very useful information here, namely:

I have also seen the error disappearing after retriggering the GHA run.

So maybe there's some kind of state that is available to subsequent executions that isn't available to the first one in a fresh environment?

This is the context reported by @MauroLuzzatto:

So far, I have only seen the errors on

• python:3.8-slim-bullseye
• Ubuntu: 22.04.2 LTS

And not in any local development environment:

• macOS Ventura Version 13.3.1

However, that might also be connected to the fact that the dbt and other modules are installed there (CI and deployment) on every run.

[emmyoop]

We should replace sqlparse with a manual solution.

[dbeatty10]

💡 @emmyoop do we have an issue in GitHub to replace sqlparse with a manual solution? Or would that issue need to be created?

[emmyoop]

Closing in favor of #7791

[jtcohen6]

Reopening per #7791 (comment)

[bdashrad]

Looks like #7919 has pivoted back to use sqlparse 0.4.4 as well

[gshank]

New fix in #8215