Failure to build on macOS with Python 3.9.0

[fxcoudert]

What did you do?

I'm a Homebrew maintainer, trying to build Pillow as part of a Homebrew package: Homebrew/homebrew-core#62795 on macOS with Python 3.9.0. This is one example, but other packages using Pillow fail in the same way.

What did you expect to happen?

Compilation succeeds.

What actually happened?

    The headers or library files could not be found for zlib,
    a required dependency when compiling Pillow from source.

What are your OS, Python and Pillow versions?

• OS: macOS 10.13 to 10.15
• Python: 3.9.0
• Pillow: 7.2.0

[radarhere]

In two days, on October 15, Pillow 8.0 is due to be released, along with corresponding Python 3.9 wheels. Will that solve your problem?

[fxcoudert]

I have seen this merged PR: #4842
As far as I understand it, the problem is not fixed by this PR, which is simply adding /usr/local/opt/zlib/include to the list of search paths. However, for most Homebrew users, the zlib package is not installed (and this directory does not exist), because most software on macOS links to system zlib.

The system zlib headers exist, and Pillow should be finding them and using them. It's also possible for Homebrew to give some help, by setting an appropriate environment variable: is there a suitable variable that Pillow would check?

[fxcoudert]

In two days, on October 15, Pillow 8.0 is due to be released, along with corresponding Python 3.9 wheels. Will that solve your problem?

My problem is building from source, so no, the wheels will not help.

[radarhere]

The system zlib headers exist, and Pillow should be finding them and using them.

If you think this is a silly question, please bear with me - where do they exist? What path are you referring to?

[fxcoudert]

Not a silly question at all… the way that it works on recent macOS versions is very unusual. The toolchain can install multiple SDKs, with one SDK being "default", and there is nothing in /usr/include.

In 99% of cases, the zlib headers are in one of two places, namely:

/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/zlib.h
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/zlib.h

The prefix for that can be obtained by running:

$ xcrun --show-sdk-path
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk

[fxcoudert]

So I guess the solution for Pillow, if it wants to look up header files, would be to try and run xcrun --show-sdk-path, and if works add $(xcrun --show-sdk-path)/usr/include to its search path.

In the meantime, is there an environment variable we could pass to the existing 7.2.0 version to unbreak our builds?

[radarhere]

How about specifying the path through python3.9 setup.py build_ext -I/a/b install?

[fxcoudert]

Hum, often we're not calling setup.py ourselves, it's done by pip or in some Makefile. Is there a suitable PIP_INSTALL_PARAMS that could work?

[radarhere]

https://pillow.readthedocs.io/en/latest/installation.html#building-from-source

If the prerequisites are installed in the standard library locations for your machine (e.g. /usr or /usr/local), no additional configuration should be required. If they are installed in a non-standard location, you may need to configure setuptools to use those locations by editing setup.py or setup.cfg, or by adding environment variables on the command line:

CFLAGS="-I/usr/pkg/include" python3 -m pip install --upgrade Pillow

[radarhere]

I've created PR #4974 to resolve this properly. Let us know if you have any feedback on that.

[fxcoudert]

Adding CFLAGS seems to get the build past this stage. However, I get a later failure with:

    src/_imagingft.c:25:10: fatal error: 'ft2build.h' file not found
    #include <ft2build.h>
             ^~~~~~~~~~~~

We're never use freetype before for this build, and it used to pass. Does Pillow now require freetype? I would expect it to detect it and, if it does not find it, not use it.

[radarhere]

This should not be the case. I also expect Pillow to look for 'ft2build.h', and not try to include it if it is not present.

Could you provide a more complete log?

[nulano]

Could you provide a more complete log?

From the linked PR in homebrew (https://github.com/Homebrew/homebrew-core/pull/62795/checks?check_run_id=1247827911#step:5:347):

clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk -I/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -I/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/usr/include -I/opt/X11/include/freetype2 -I/usr/local/Cellar/openjpeg/2.3.1/include/openjpeg-2.3 -I/private/tmp/pip-install-i23vegwi/pillow/src/libImaging -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/private/tmp/filebeat-20201013-79912-dm5vno/src/github.com/elastic/beats/filebeat/build/python-env/include -I/opt/X11/include -I/usr/local/include -I/usr/local/include -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/sqlite/include -I/private/tmp/filebeat-20201013-79912-dm5vno/src/github.com/elastic/beats/filebeat/build/python-env/include -I/usr/local/Cellar/python@3.9/3.9.0/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c src/_imagingft.c -o build/temp.macosx-10.14-x86_64-3.9/src/_imagingft.o

It seems there might be a version of FreeType in /opt/X11/include/freetype2.

[nulano]

It would be nice to compare this log with a previous build to see if that did in fact use FreeType, but it seems brew doesn't print logs for successful runs (e.g. https://github.com/Homebrew/homebrew-core/pull/61603/checks?check_run_id=1161020342#step:5:36).

[radarhere]

Your current PR - Homebrew/homebrew-core#63381 - uses Pillow 7.0. Would you be able to upgrade that to 8.0.x? In addition to being the first release series that we've tested for Python 3.9, it would then include #4974 and #4842, which could help?

[julianrinaldi]

I have the same issue listed here when trying to install Home Assistant. Is there a way to fix this issue yet?

[radarhere]

@julianrinaldi I presume you're talking about a problem with zlib? This issue has also discussed a problem with freetype.

#4974 should have helped this by searching through the SDK paths as well. If this is still happening for you on macOS with the latest version of Pillow, could you create a new issue with more details?

[julianrinaldi]

I was able to fix it by installing zlib through brew and following the directions to point it to the right location. Thanks.

[radarhere]

@fxcoudert I see you've closed Homebrew/homebrew-core#63381. Are you no longer interested in this?

[fxcoudert]

I don't have the time to devote to finding and applying a fix. I would welcome a pull request to Homebrew, and an improvement of the Pillow build system so that it works better on macOS.

[radarhere]

Each of the formulas from your PR have now been updated to Python 3.9

• auditbeat: Homebrew/homebrew-core@1c8b266
• filebeat: Homebrew/homebrew-core@18270cd
• heartbeat: Homebrew/homebrew-core@deb0426
• packetbeat: Homebrew/homebrew-core@6b4da3a

Does that mean that this issue is now resolved?