psycopg2-binary support for M1 chipset

[armenzg]

Hello all,
With a big of guidance I'm happy to contribute a PR.
For now, let me document the current situation for any other M1 users. Please do correct me if I got the details wrong!

Python 3.8.10 is the first 3.8 version with support for M1 chipset (see bug). Older versions of pyenv do not have access to Python 3.8.10 (some details in here).

pyenv install 3.8.10
python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip wheel

If you try to install psycopg2_binary and expect a wheel, it is not currently available.

There's the SYSTEM_VERSION_COMPAT=1 trick to pretend to be 10.6 instead of 11.x, however, the architecture still returns arm64.

❯ python -c "import platform; print(platform.mac_ver())"
('11.4', ('', '', ''), 'arm64')

~/code/M1 via 🐍 v3.8.2 (venv_3_system) on ☁️  armenzg@sentry.io
❯ SYSTEM_VERSION_COMPAT=1 python -c "import platform; print(platform.mac_ver())"
('10.16', ('', '', ''), 'arm64')

If you don't have a wheel then you need to build from source. See the log [1] at the bottom of this comment.

In order to have the libraries to build it, I took these steps which are based on this post:

brew install postgresql
# If you open a new terminal tab you will see that pg_config is available
export CPPFLAGS="-I/opt/homebrew/opt/openssl@1.1/include"
export LDFLAGS="-L/opt/homebrew/opt/openssl@1.1/lib -L${HOME}/.pyenv/versions/3.8.10/lib"

After that, it is possible to build the wheel

pip install psycopg2-binary==2.8.6
Collecting psycopg2-binary==2.8.6
  Using cached psycopg2-binary-2.8.6.tar.gz (384 kB)
Building wheels for collected packages: psycopg2-binary
  Building wheel for psycopg2-binary (setup.py) ... done
  Created wheel for psycopg2-binary: filename=psycopg2_binary-2.8.6-cp38-cp38-macosx_11_0_arm64.whl size=137096 sha256=af1103eb3c4d9fee1636d27e9dbab6c8bbe7fcd8502bea2a4a194f016a8afb85
  Stored in directory: /Users/armenzg/Library/Caches/pip/wheels/15/d7/13/ef37b82cf1de521c82c129a16ad8007a81b4eab0f23b22c7b3
Successfully built psycopg2-binary
Installing collected packages: psycopg2-binary
Successfully installed psycopg2-binary-2.8.6

Did I do this right? How can I help to have a release that includes an arm64 wheel?

[1]

SYSTEM_VERSION_COMPAT=1 pip install psycopg2-binary==2.8.6
Collecting psycopg2-binary==2.8.6
  Using cached psycopg2-binary-2.8.6.tar.gz (384 kB)
    ERROR: Command errored out with exit status 1:
     command: /Users/armenzg/code/M1/venv_3_system/bin/python3 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-9zonoxgp/psycopg2-binary_43fe9c32f13f470ebebbbdf290cabfe4/setup.py'"'"'; __file__='"'"'/private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-9zonoxgp/psycopg2-binary_43fe9c32f13f470ebebbbdf290cabfe4/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-pip-egg-info-cacpi1hg
         cwd: /private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-install-9zonoxgp/psycopg2-binary_43fe9c32f13f470ebebbbdf290cabfe4/
    Complete output (23 lines):
    running egg_info
    creating /private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-pip-egg-info-cacpi1hg/psycopg2_binary.egg-info
    writing /private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-pip-egg-info-cacpi1hg/psycopg2_binary.egg-info/PKG-INFO
    writing dependency_links to /private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-pip-egg-info-cacpi1hg/psycopg2_binary.egg-info/dependency_links.txt
    writing top-level names to /private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-pip-egg-info-cacpi1hg/psycopg2_binary.egg-info/top_level.txt
    writing manifest file '/private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-pip-egg-info-cacpi1hg/psycopg2_binary.egg-info/SOURCES.txt'

    Error: pg_config executable not found.

    pg_config is required to build psycopg2 from source.  Please add the directory
    containing pg_config to the $PATH or specify the full executable path with the
    option:

        python setup.py build_ext --pg-config /path/to/pg_config build ...

    or with the pg_config option in 'setup.cfg'.

    If you prefer to avoid building psycopg2 from source, please install the PyPI
    'psycopg2-binary' package instead.

    For further information please check the 'doc/src/install.rst' file (also at
    <https://www.psycopg.org/docs/install.html>).

    ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/fc/51/0f2c6aec5c59e5640f507b59567f63b9d73a9317898810b4db311da32dfc/psycopg2-binary-2.8.6.tar.gz#sha256=11b9c0ebce097180129e422379b824ae21c8f2a6596b159c7659e2e5a00e1aa0 (from https://pypi.org/simple/psycopg2-binary/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement psycopg2-binary==2.8.6 (from versions: 2.7.4, 2.7.5, 2.7.6, 2.7.6.1, 2.7.7, 2.8, 2.8.1, 2.8.2, 2.8.3, 2.8.4, 2.8.5, 2.8.6)
ERROR: No matching distribution found for psycopg2-binary==2.8.6

[dvarrazzo]

Hello @armenzg

A change to fix the CPPFLAGS/LDFLAGS has already landed in master (52cd944), so that will not be needed anymore from next release on.

Are you able to fix https://github.com/psycopg/psycopg2/blob/master/scripts/build/build_macos.sh in order to build wheels working on arm too?

Thank you!

[armenzg]

Hello @armenzg

A change to fix the CPPFLAGS/LDFLAGS has already landed in master (52cd944), so that will not be needed anymore from next release on.

Are you able to fix https://github.com/psycopg/psycopg2/blob/master/scripts/build/build_macos.sh in order to build wheels working on arm too?

Thank you!

Hi Daniel,
With regards to helping I'm more likely able to help end of July or beginning of August as I wrap my current top goals.

Do you (or anyone reading this later) know if you need an Arm device in order to build the wheel? Does this mean that there would be need for Arm infra? (I don't think GH has such kind of runners; maybe require a custom runner?) Or a developer building it in their machine and transferring the wheel to whoever can release it?

I'm very unfamiliar with wheels, thus, my questions might be rather naive.

Best wishes,
Armen

[dvarrazzo]

I am unfamiliar with the M1 chipset so I might be naive as well. But I believe the Mac OS toolkit can cross-compile.

[pedromassango]

Hello. Could anyone tell how to get this paackage to work on a M1 machine? What are the step-by-step workarounds? Thank you!

[armenzg]

@pedromassango Hi Pedro,
This worked for me:

I took these steps which are based on this post:

❯ brew install postgresql
# If you open a new terminal tab you will see that pg_config is available
❯ export CPPFLAGS="-I/opt/homebrew/opt/openssl@1.1/include"
❯ export LDFLAGS="-L/opt/homebrew/opt/openssl@1.1/lib -L${HOME}/.pyenv/versions/3.8.10/lib"
❯ python -V
Python 3.8.10
❯ pip install psycopg2-binary==2.8.6

[armenzg]

@pedromassango you will not need the CPPFLAGS & LDFLAGS if you install a newer psycopg2-binary than 2.8.6.

[armenzg]

I will be away for few months starting in November so I don't know if I will be able to contribute a fully tested script, however, I have a draft in here:
https://gist.github.com/armenzg/7fc4394a1d2b5bb44b31902d1a3e6777#file-build_macos-sh

I uploaded the produced wheel to a Google Cloud storage that I make public. Other engineers can pip install it on their M1 machines w/o having to install postgresql.

[simkimsia]

i also figured out how to install without postgresql.

My setup is for macOS monterey 12.1.0 and using venv i got pg_config not found with python 3.9.9

My answer is here at StackOverflow https://stackoverflow.com/a/70482055/80353

[fikisipi]

I've uploaded a binary here: https://github.com/fikisipi/psycopg2-m1

[dvarrazzo]

@fikisipi any reason why you don't contribute a merge request to this project?

[fikisipi]

@dvarrazzo I'd like to, but you'll need to set up a private GitHub Action runner in the repo settings.

In my repo I booted an Action Runner on my own machine as you can see in the job run log, but for trust purposes (unless the build is deterministic and a checksum is enough?) I think we should rent a scaleway m1 instance for a day. We can further use the native Action to provision the m1 runner that will produce the binary in a transparent way.

[dvarrazzo]

@fikisipi Trust is an important matter, yes, so I don't think it is wise to distribute packages in the project name which are built by third parties.

I don't doubt about your good faith or intention but I wouldn't want to create a precedent for others, while at the same time desensitise people into no longer auditing the quality of the packages they install.

Recent events of the guy who intentionally messed up the npm packages he maintained should be a warning call for users and a responsibility call for package authors.

As such, I would kindly ask you to request pypi to withdraw the package you distributed. I'm happy to look together with you into how to build these packages reliably and distribute them as psycopg2-binary. There is also the problem that it's impossible to create a portable requirements file if the distribution package name is different - a problem we already have with the psycopg2/psycopg2-binary packages. I don't wish to make this more complex for users.

Thank you for your understanding and please know that I appreciate your gesture :)

[fikisipi]

@dvarrazzo I removed the pypi release, your arguments are completely valid 👍

Would you like me to create a pull request here with a GitHub action for provisioning a M1 build job running on Scaleway
or AWS (the only cloud platforms offering M1)? If done correctly, it should be tamper-proof. The only issue is that Apple enforces a minimum 24 hour allocation per customer for Mac SaaS platforms, which means a minimum of ~$2 to start building (and assuming the psycopg2 release cycle has a time window of >= 24 hours between versions, it also means $2 per release).

[dvarrazzo]

Hello @fikisipi

Thank you very much for your understanding: I appreciate it.

I got in touch with Scaleway and asked if they would be available to sponsor the Psycopg project by offering us their service in order to build the packages, otherwise to give us a quote for their service.

I assume we can foot a bill of $2 per release. I know well that Apple is short on petty cash and they absolutely need the money from our project to survive, I am very sympathetic of their cause.

If, in turn, some of the users interested in the M1 packages would like to contribute to 💜 sponsor the project, it would be great. Thank you very much!

[fikisipi]

@dvarrazzo Looking at it, pypa/cibuildwheel seems to have some kind of cross-compilation command for Apple Silicon:

https://github.com/pypa/cibuildwheel/blob/1d2f86e6cf7fe43faef886d32748d6904c0e3a02/cibuildwheel/macos.py#L413-L423

I'll try that as well. I see @armenzg tried that in his fork but I can't see if there has been progress.

[dvarrazzo]

I have crated some test binary packages for Apple M1.

Please test them using:

pip install -i https://test.pypi.org/simple/ psycopg2-binary==2.9.3

and verify that they work, e.g.

python3
>>> import psycopg2
>>> psycopg2.__version__
'2.9.3 (dt dec pq3 ext lo64)'
>>> conn = psycopg2.connect("dbname=postgres")
>>> cur = conn.cursor()
>>> cur.execute("select 1+1")
>>> print(cur.fetchone()[0])
2

[stephane-klein]

I have crated some test binary packages for Apple M1.

Please test them using:

pip install -i https://test.pypi.org/simple/ psycopg2-binary==2.9.3

Thanks 👍, this package also works on my M1.

Can I use temporary this "test" version in my projectrequirements.txt?

@dvarrazzo is it planned to fix https://pypi.org/project/psycopg2-binary/#files version ?

[dvarrazzo]

@stephane-klein thank you for testing. Yes, of course the plan is to make these wheels available on PyPI.

[jabruder]

The test binary package you've made for Apple silicon seem to be working well for me. Thank you!

[dfioravanti]

Yes, of course the plan is to make these wheels available on PyPI.

Hi everyone, is there a timeline for when will this happen? The fact that I cannot install the binary is really annoying right now. If there is something to do that just takes time and not a lot of knowledge in psycopg2 I am more than happy to help with it 😊

[takecare]

@stephane-klein have you been able to use it on your requirements file? i've tried it in a couple of ways - e.g. psycopg2-binary @ https://test.pypi.org/project/psycopg2-binary/2.9.3/ - but i get an error like the following:

ERROR: Cannot unpack file /private/var/folders/9s/73tc980977s8fyxv4d14rkjc0000gn/T/pip-unpack-0m67uu6j/simple (downloaded from /private/var/folders/9s/73tc980977s8fyxv4d14rkjc0000gn/T/pip-install-f_gsozn9/psycopg2-binary_9f2f57177fba4610bc56018ad5f88b0d, content-type: text/html; charset=UTF-8); cannot detect archive format
ERROR: Cannot determine archive format of /private/var/folders/9s/73tc980977s8fyxv4d14rkjc0000gn/T/pip-install-f_gsozn9/psycopg2-binary_9f2f57177fba4610bc56018ad5f88b0d

pip 22.2.1 (python 3.10.5).

it works fine when installing manually (as suggested above).

[dvarrazzo]

The build scripts are merged to master.

For information about the packages release please see #1482.

[npbgia]

I have crated some test binary packages for Apple M1.

Please test them using:

pip install -i https://test.pypi.org/simple/ psycopg2-binary==2.9.3

and verify that they work, e.g.

python3
>>> import psycopg2
>>> psycopg2.__version__
'2.9.3 (dt dec pq3 ext lo64)'
>>> conn = psycopg2.connect("dbname=postgres")
>>> cur = conn.cursor()
>>> cur.execute("select 1+1")
>>> print(cur.fetchone()[0])
2

This is the unique way helped me resolved this issue on my Mac M1, I have google searched for a week kakaka
Thank you a lot, you save my life !!!

[Eugene-Lam]

I have crated some test binary packages for Apple M1.

Please test them using:

pip install -i https://test.pypi.org/simple/ psycopg2-binary==2.9.3

and verify that they work, e.g.

python3
>>> import psycopg2
>>> psycopg2.__version__
'2.9.3 (dt dec pq3 ext lo64)'
>>> conn = psycopg2.connect("dbname=postgres")
>>> cur = conn.cursor()
>>> cur.execute("select 1+1")
>>> print(cur.fetchone()[0])
2

Thx a lot. It solves my issue.

[fikisipi]

@dvarrazzo Works well. Hope to see it in prod soon.

[tekumara]

@takecare try specifying the wheel in requirements.txt, eg: for python 3.9

psycopg2-binary @ https://test-files.pythonhosted.org/packages/be/5f/c3c829beda843fc6c75ccd0b428a9ee04346400adc7d269709fd536c22ce/psycopg2_binary-2.9.3-cp39-cp39-macosx_11_0_arm64.whl ; platform_system == 'Darwin' and platform_machine == 'arm64' and python_version == '3.9'

[gnuletik]

Works great thanks!

For poetry users, you can:

Add test-pypi as a secondary source

diff --git a/pyproject.toml b/pyproject.toml
index a886bbd..194f07f 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -5,13 +5,18 @@
 description = ""
 license = "MIT"

+[[tool.poetry.source]]
+name = "test-pypi"
+url = "https://test.pypi.org/simple"
+secondary = true
+
 [tool.poetry.dependencies]
 python = "^3.10"

And install from test-pypi

poetry add psycopg2-binary@latest --source test-pypi

[tomzorz]

thanks for the poetry solve @gnuletik 💯

[pythoninthegrass]

And install from test-pypi

poetry add psycopg2-binary@latest --source test-pypi

This worked for me as well. Do we still need the test-pypi source or has it made it into pypi proper yet?

[abrigard]

I installed using:
arch -arm64 pip3 install -i https://test.pypi.org/simple/ psycopg2-binary==2.9.3
And then when executing the python file:
arch -arm64 /env-path/python3 pythonfile.py