[Bug]: Python builds fail with google-api-core dep version cannot be satisfied

[Abacn]

What happened?

Python jenkin jobs that requires install apache-beam[gcp] all fail.

Issue Priority

Priority: 0

Issue Component

Component: build-system

[Abacn]

There is another dep error coming from atomicwrites, also fixing it in this issue

[Abacn]

Per #22218 this also affects released version installation. A workaround is to install google-cloud-bigquery-storage<2.14 prior to installing apache-beam. For docker file, e.g.

RUN pip install google-cloud-bigquery-storage==2.13.2
RUN pip install apache-beam[gcp]==2.40.0

[Abacn]

Needs reopen because the rollout of google-cloud-pubsub 2.13.2 triggered the exactly same problem. Should consider #22208 as long term fix

[robertwb]

How severe is this? Do we need a point release? Or will this be solved by upstream updates (that are hopefully in progress)?

[Abacn]

How severe is this? Do we need a point release? Or will this be solved by upstream updates (that are hopefully in progress)?

It affects all released versions. Per #22218 (comment) "currently causing fairly severe issues".

imo this should be resolved upstream. We can only provide workaround for installation (manually install some sub-latest deps before pypi install apache-beam). The dep requirements itself does not have problems, but due to upstream client lib bumping api-core major version in there patch release we unfortunately hit a pypi resolving issue/bug

[robertwb]

So, pretty much anyone who does pip install apache-beam[gcp] will encounter a failure?

[Abacn]

So, pretty much anyone who does pip install apache-beam[gcp] will encounter a failure?

Yes, if their environment does not have compatible google-cloud-* client packages already installed.
THen pypi will try to install these deps and go endless backtracking

[tvalentyn]

Let's use #22218 for further tracking which now has more discussion and workarounds.