[Bug]: Dependency resolution is happening when installing latest version of apache-beam[gcp]

[tvalentyn]

What happened?

Installing apache-beam into a clean environment now shows following logs:

 pip install --no-cache-dir --pre apache-beam[gcp]==2.56.0
...

Collecting grpcio-status>=1.33.2 (from google-cloud-pubsub<3,>=2.1.0->apache-beam[gcp]==2.56.0)
  Downloading grpcio_status-1.64.1-py3-none-any.whl.metadata (1.1 kB)
  Downloading grpcio_status-1.64.0-py3-none-any.whl.metadata (1.1 kB)
  Downloading grpcio_status-1.64.0rc1-py3-none-any.whl.metadata (1.1 kB)
  Downloading grpcio_status-1.63.0-py3-none-any.whl.metadata (1.1 kB)
  Downloading grpcio_status-1.63.0rc2-py3-none-any.whl.metadata (1.1 kB)
  Downloading grpcio_status-1.63.0rc1-py3-none-any.whl.metadata (1.1 kB)

That means, when analyzing dependency ranges, pip cannot take the latest allowed version for each dependency, and has to do dependency resolution. This is suboptimal and can result in failed installations (https://s.apache.org/beam-python-dependencies-pm). We should check what combination of packages requires the resolution and work with maintainers of Beam dependencies to remove the need for resolution if possible.

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[tvalentyn]

* grpcio-status==1.66.1
 - protobuf [required: >=5.26.1,<6.0dev, installed: 4.25.5]

Should be fixed by switching to protobuf >=5 or mitigated by
adding a bound "grpcio-status<1.63.0"

[github-actions]

This issue has been marked as stale due to 150 days of inactivity. It will be closed in 30 days if no further activity occurs. If you think that’s incorrect or this issue still needs to be addressed, please simply write any comment. If closed, you can reopen the issue at any time. Thank you for your contributions.

[github-actions]

This issue has been closed due to lack of activity. If you think that is incorrect, you can reopen the issue at any time.

[tvalentyn]

Should be fixed by switching to protobuf >=5

Looks like this is still happening even with protobuf 5 being allowed, most likely due to #34081 . We can confirm that this is the culprint. We expect the fix for it to be available early 2026, if not we can proceed with:

or mitigated by adding a bound "grpcio-status<1.63.0"

until fix is available

[tvalentyn]

We can confirm that https://github.com/apache/beam/issues/34081is the culprint.

Installing:

pip install apache-beam[gcp] --use-deprecated=legacy-resolver

Gives:

grpcio-status 1.76.0 requires grpcio>=1.76.0, but you'll have grpcio 1.65.5 which is incompatible.

[tvalentyn]

so yes, once we can upgrade to newer GRPC, this shouldn't be an issue.