Skip to content

Conversation

@ashb
Copy link
Member

@ashb ashb commented Nov 4, 2025

This was pulling in psycopg2, but out own postres provider uses
psycopg2-binary which is essentially in conflict.

Since this wasn't behind an extra, and not every user needs it, and since it
conflicts with our own postgres deps, I think removal is the best option.

This was only recently added/made working in #57141

The error this caused was on my local machine where pg_config is not installed:

uv run -p 3.12 python -m pytest airflow-core/tests/unit/jobs/test_scheduler_job.py
Using CPython 3.12.7
Removed virtual environment at: .venv
Creating virtual environment at: .venv
  × Failed to build `psycopg2==2.9.11`
  ├─▶ The build backend returned an error
  ╰─▶ Call to `setuptools.build_meta:__legacy__.build_wheel` failed (exit status: 1)

      [stdout]
      running egg_info
      writing psycopg2.egg-info/PKG-INFO
      writing dependency_links to psycopg2.egg-info/dependency_links.txt
      writing top-level names to psycopg2.egg-info/top_level.txt

      [stderr]
      /Users/ash/.cache/uv/builds-v0/.tmpkOwkPL/lib/python3.12/site-packages/setuptools/dist.py:759: SetuptoolsDeprecationWarning: License classifiers are deprecated.
      !!

              ********************************************************************************
              Please consider removing the following classifiers in favor of a SPDX license expression:

              License :: OSI Approved :: GNU Library or Lesser General Public License (LGPL)

              See https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#license for details.
              ********************************************************************************
      !!
        self._finalize_license_expression()

      Error: pg_config executable not found.
psycopg2` (v2.9.11) was included because `apache-airflow[all]` (v3.2.0) depends on `apache-airflow-providers-weaviate` (v3.2.4) which depends on `pandas[postgresql]` (v2.1.4) which depends on `psycopg2`

(That last part was actually incorrect, it is not weaviate that depended on the extra, but presto)

@ashb ashb force-pushed the remove-postgres-extra-presto branch from a6e1069 to 060f0e3 Compare November 4, 2025 16:14
@ashb ashb requested a review from potiuk November 4, 2025 16:15
@ashb ashb force-pushed the remove-postgres-extra-presto branch from 060f0e3 to 0fc9068 Compare November 4, 2025 16:33
@potiuk potiuk force-pushed the remove-postgres-extra-presto branch from 0fc9068 to 2a1f933 Compare November 4, 2025 18:44
@potiuk
Copy link
Member

potiuk commented Nov 4, 2025

Rebased .. we had two cross-merged PRs that broke main

This was pulling in psycopg2, but out own postres provider uses
psycopg2-binary which is essentially in conflict.

Since this wasn't behind an extra, and not every user needs it, _and_ since it
conflicts with our own postgres deps, I think removal is the best option.

This was only recently added/made working in #57141
@ashb ashb force-pushed the remove-postgres-extra-presto branch from 2a1f933 to b42f6bf Compare November 4, 2025 22:00
@ashb ashb changed the title Remove the "postgres" extra on pandas from Presto provider entirely. Replace the "postgres" extra on pandas from Presto provider with direct psycopg2-binary Nov 5, 2025
@ashb ashb merged commit e50a337 into main Nov 5, 2025
94 checks passed
@ashb ashb deleted the remove-postgres-extra-presto branch November 5, 2025 10:01
Copilot AI pushed a commit to jason810496/airflow that referenced this pull request Dec 5, 2025
…ct psycopg2-binary. (apache#57813)

This was pulling in psycopg2, but out own postres provider uses
psycopg2-binary which is essentially in conflict.

Since this wasn't behind an extra, and not every user needs it, _and_ since it
conflicts with our own postgres deps, I think removal is the best option.

This was only recently added/made working in apache#57141 (before then it issued a
warning about an unknown extra)

But some things have an implicit dep on Psycopg2 being available, so we
depend directly on the (correct for us) psycopg2-binary.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants