[CT-1363] [Bug] 1.3.0 has inconsistent dependencies

[judahrand]

Is this a new bug in dbt-bigquery?

• I believe this is a new bug in dbt-bigquery
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

requirements.in

dbt-bigquery~=1.3

pip-compile requirements.in results in:

There are incompatible versions in the resolved dependencies:
  protobuf<4.0.0dev,>=3.12.0 (from google-cloud-bigquery==2.34.4->dbt-bigquery==1.3.0->-r requirements/prod.in (line 1))
  protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 (from google-cloud-dataproc==5.0.3->dbt-bigquery==1.3.0->-r requirements/prod.in (line 1))
  protobuf<4,>=3.13.0 (from dbt-bigquery==1.3.0->-r requirements/prod.in (line 1))
  protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 (from google-api-core[grpc]==2.10.2->dbt-bigquery==1.3.0->-r requirements/prod.in (line 1))
  protobuf>=4.21.3 (from grpcio-status==1.49.1->google-api-core[grpc]==2.10.2->dbt-bigquery==1.3.0->-r requirements/prod.in (line 1))
  protobuf<5.0.0dev,>=3.15.0 (from googleapis-common-protos==1.56.4->dbt-bigquery==1.3.0->-r requirements/prod.in (line 1))
  protobuf<5.0.0dev,>=3.19.0 (from proto-plus==1.22.1->google-cloud-dataproc==5.0.3->dbt-bigquery==1.3.0->-r requirements/prod.in (line 1))

The issue appears to be this line:

  protobuf>=4.21.3 (from grpcio-status==1.49.1->google-api-core[grpc]==2.10.2->dbt-bigquery==1.3.0->-r requirements/prod.in (line 1))

Expected Behavior

A consistent set of dependencies can be found

Steps To Reproduce

See current behaviour.

Relevant log output

No response

Environment

- OS: MacOS 12.6 
- Python: 3.10
- dbt-core: NA
- dbt-bigquery: NA

Additional Context

No response

[jtcohen6]

We've seen issues in the past where pip install dbt-* works perfectly fine (as it does for me locally, running pip install dbt-bigquery~=1.3 in a fresh virtualenv), but pip-compile finds conflicts: dbt-labs/dbt-core#2674, dbt-labs/dbt-core#2676

We don't do any testing against pip-compile, and we aren't able to guarantee that it will work. We do regularly test to ensure that pip install dbt-* works for all officially supported plugins, OS, and Python versions.

[judahrand]

Related but not exactly this issue, @jtcohen6, is it wise to pin Jinja2 to a specific version? Would it not be safer to pin to Jinja2~=3.1.2 instead of Jinja2==3.1.2? Pinning to a specific version is much more likely to cause these sorts of issues.

[mhindery]

We are having issues with dependancies as well, in our case using dbt-postgres but the issue is exactly the same. Specifically we're installing dbt next to some other packages, and a pip install does not work anymore: it fails to select suitable versions of dependancies and times out eventually. We've tried to use pip-tools to pin versions of dependancies, but encountered similar issues as described above: dbt itself has a conflicting set of dependancies so it cannot be used with pip-tools, regardless of what we bundle it with.

On top of that I agree with the remark just above me: the required versions of dependancies are way to specific. This makes it near impossible to install dbt side by side with other packages. The versions could be relaxed to compatible minor versions with some boundaries rather than desiring an exact patch release.

I don't agree with the response from the authors here: your desired set of dependancies is inherently broken, you don't actually know what is installed using a pip install. It is up to pip to select 'something' if it can, which might or might not work and is not under your control. You're asking for incompatible constraints, which should not be an acceptable situation to start with.

[jtcohen6]

We've experienced breaking changes due to Jinja2 patch releases in the past (dbt-labs/dbt-core#2318 (comment)). Jinja is essential to a lot of user-space code, which is why we pin it so aggressively.

We regularly test to ensure that pip install dbt-* works for all our supported adapters. If any of them stops working, it's an incident that we work to resolve ASAP. We can't test those installations for compatibility with every other possible dependency, though.

At the same time, I hear the concerns, and I want to see us moving toward the future I described in dbt-labs/dbt-core#4748 (comment):

• Looser dependency pins in setup.py
• Exact (==) dependency versions in a requirements.txt, which we compile + test + distribute with every dbt-core release, and for which we can guarantee total working functionality

The work ahead of us is:

• Defining that policy in a central, easily reference-able place ([Draft] Adding dependency docs and requirements.txt dbt-core#5450)
• Making it into a reality

[Tenzer]

See also #361 where I can't install dbt-bigquery with pip install dbt-bigquery==1.3.0.