Installing psycopg2-binary with Python:3.6.4-alpine doesn't work

[barrachri]

When you try to install the binary version of the lib you get the source version error.

$ docker run -it --rm python:3.6.4-alpine sh
$ pip install psycopg2-binary

Collecting psycopg2-binary
  Downloading psycopg2-binary-2.7.4.tar.gz (426kB)
    100% |████████████████████████████████| 430kB 52kB/s
    Complete output from command python setup.py egg_info:
    running egg_info
    creating pip-egg-info/psycopg2_binary.egg-info
    writing pip-egg-info/psycopg2_binary.egg-info/PKG-INFO
    writing dependency_links to pip-egg-info/psycopg2_binary.egg-info/dependency_links.txt
    writing top-level names to pip-egg-info/psycopg2_binary.egg-info/top_level.txt
    writing manifest file 'pip-egg-info/psycopg2_binary.egg-info/SOURCES.txt'

    Error: pg_config executable not found.

    pg_config is required to build psycopg2 from source.  Please add the directory
    containing pg_config to the $PATH or specify the full executable path with the
    option:

        python setup.py build_ext --pg-config /path/to/pg_config build ...

    or with the pg_config option in 'setup.cfg'.

    If you prefer to avoid building psycopg2 from source, please install the PyPI
    'psycopg2-binary' package instead.

    For further information please check the 'doc/src/install.rst' file (also at
    <http://initd.org/psycopg/docs/install.html>).

[dvarrazzo]

Alpine Linux use musl libc and manylinux wheels don't support it: see e.g. pypa/manylinux#37. Not a psycopg problem but a lack of a standard for it, such as PEP 513 defininig manylinux1. Happy to support Alpine as soon as there will be a standard and tools for it.

[nourspace]

I found this solution which worked quite fine.

FROM python:3.6-alpine

RUN apk update && apk add postgresql-dev gcc python3-dev musl-dev

pip install psycopg2-binary

---

Edit

The above compiles the package from source, so I would rather

FROM python:3.6-alpine
RUN apk update && apk add postgresql-dev gcc python3-dev musl-dev
pip install psycopg2

not to get confused.

[barrachri]

But I think you are pseudo-using the binary package and building from the source at the end.

[dvarrazzo]

@nourchawich your "solution" is about compiling psycopg from source, which is totally supported. Binary packages are not (for the reason described above).

[nourspace]

Thank for the clarification @barrachri and @dvarrazzo

I just noticed what was actually happening. All I wanted was to have psycopg2 (regardless whether binary or not) installed on an alpine based image, hence considering it a solution.

[cscetbon]

@nourchawich after it's been built, I suppose we can get rid of those dev packages ?

[siaarzh]

@nourchawich after it's been built, I suppose we can get rid of those dev packages ?

Yes, but make sure the apt add and apt del commands are in one layer. Otherwise you won't make the image any smaller.
E.g.:

RUN apk add --no-cache --virtual .build-deps \
    gcc \
    python3-dev \
    musl-dev \
    postgresql-dev \
    && pip install --no-cache-dir psycopg2 \
    && apk del --no-cache .build-deps

[cscetbon]

Yeah I did, thanks !

[Cerebus]

Since this is the top google hit for this problem, use a multi-stage build to minimize the final image size.

FROM python:3.6-alpine as base                                                                                                
                                                                                                                              
FROM base as builder                                                                                                          
                                                                                                                              
RUN mkdir /install                                                                                                            
RUN apk update && apk add postgresql-dev gcc python3-dev musl-dev                                                             
WORKDIR /install                                                                                                              
COPY requirements.txt /requirements.txt                                                                                       
RUN pip install --install-option="--prefix=/install" -r /requirements.txt                                                     
                                                                                                                              
FROM base                                                                                                                     
                                                                                                                              
COPY --from=builder /install /usr/local                                                                                       
COPY src /app                                                                                                         
RUN apk --no-cache add libpq                                                                                                  
WORKDIR /app

[hugobuddel]

This is still a bit unclear to me. The point of installing psycopg2-binary is that compilers like gcc etc. are not necessary right?

I'm getting the same message as the original poster. When trying to install psycopg2-binary, it complains about not finding gcc and suggests to, well, install psycopg2-binary (which is what I'm trying to do).

[hugobuddel]

I'm on Centos 7.4, and apparently I had to yum install postgresql-devel to make psycopg2-binary to work.

[Cerebus]

@hugobuddel The problem is when pip decides it can't install the psycopg2-binary wheel it falls back on building from source, which naturally requires dev packages. On alpine that's because of muslc instead of glibc. I should think this shouldn't happen on CentOS but maybe there's another conflict.

[Pk13055]

Building on @Cerebus's answer, to get this to work with pipenv, you can try something like this:

FROM python:3.6-alpine as base

FROM base as builder

RUN mkdir /install
RUN apk update \
 && apk add postgresql-dev gcc python3-dev musl-dev
RUN pip install pipenv  # to lock and create requirements.txt
WORKDIR /install

# creation of requirements.txt
COPY Pipfile ./Pipfile
RUN pipenv lock -r > requirements.txt

# everything else stays the same
RUN pip install --install-option="--prefix=/install" -r ./requirements.txt

FROM base

COPY --from=builder /install /usr/local
COPY .  /app
RUN apk --no-cache add libpq
WORKDIR /app

[nourspace]

While it is possible to use multi-stage builds or add libraries to compile from source, I recently started using the slim version instead of alpine.

It installs psycopg2-binary without issues.

FROM python:3.6-slim

pip install psycopg2-binary

I found this article to be very useful in deciding which Python base image to use for production.
https://pythonspeed.com/articles/base-image-python-docker-images/

[finete]

@Pk13055 I would avoid using pipenv as it seems to be dead

[Pk13055]

Haha, I have also shifted to pip-tools, much cleaner and faster!

…

On Wed, Mar 4, 2020 at 2:24 PM MoVai ***@***.***> wrote: @Pk13055 <https://github.com/Pk13055> I would avoid using pipenv as it seems to be dead <pypa/pipenv#4058> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#684?email_source=notifications&email_token=ADG23DA7NYD4ZAYDIEC22NLRFYJLZA5CNFSM4ESZDXMKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENW5AXA#issuecomment-594399324>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADG23DCHUYRUGZNEUWVTXDDRFYJLZANCNFSM4ESZDXMA> .

-- *Regards* Pratik Kamble

[Asday]

@MosheVai pipenv appears to have pulled a Lazarus as of a few hours ago.

[Budgiebrain994]

Alpine Linux use musl libc and manylinux wheels don't support it: see e.g. pypa/manylinux#37. Not a psycopg problem but a lack of a standard for it, such as PEP 513 defininig manylinux1. Happy to support Alpine as soon as there will be a standard and tools for it.

Looks like musllinux support has been added to manylinux: pypa/manylinux#1135

Perhaps this issue could have a revisit?

[LukasWestholt]

FROM python:3.10.4-alpine
RUN apk add --no-cache --virtual build-dependencies libpq-dev build-base \
    && pip install -r requirements.txt \
    && apk del --no-cache build-dependencies

seems to work too.

[vaibhavgupta082]

not able to install psycopg2 on my docker environment

requirement.txt
Django>=3.2.4,<3.3 djangorestframework>=3.12.4,<3.13 psycopg2>=2.8.6,<2.9

dockerfile
`
FROM python:3.9-alpine3.13
LABEL maintainer="*******************"

ENV PYTHONUNBUFFERED 1

COPY ./requirements.txt /tmp/requirements.txt
COPY ./requirements.dev.txt /tmp/requirements.dev.txt
COPY ./app /app
WORKDIR /app
EXPOSE 8000

ARG DEV=false
RUN python -m venv /py &&
/py/bin/pip install --upgrade pip &&
apk add --update --no-cache postgresql-client &&
apk add --update --no-cache --virtual .tmp-build-deps
build-base postgresql-dev gcc python3-dev musl-dev &&
/py/bin/pip install -r /tmp/requirements.txt &&
if [ $DEV = "true" ];
then /py/bin/pip install -r /tmp/requirements.dev.txt ;
fi && \
rm -rf /tmp &&
apk del .tmp-build-deps &&
adduser
--disabled-password
--no-create-home
django-user

ENV PATH="/py/bin:$PATH"

USER django-user
`