Corrupt JPEG data: premature end of data segment

[tomgoddard]

Pillow version 10.1.0 in Python 3.11.6 on macOS 14.2.1 with ARM CPU writes corrupted JPEG files about 1/3 of the time. I've attached a zip file giving an example. The zip file contains 3 PNG images and a Python script gen.py that opens the PNG files using Pillow and saves them as JPEG images. ImageMagick 7.1.1 then reports 2 of the 3 JPEG files are corrupted.

$ unzip jpg_test.zip
$ ls
145.png		146.png		147.png		gen.py
$ python3 gen.py
$ ls
145.jpg		146.jpg		147.jpg		gen.py
145.png		146.png		147.png
$ identify -verbose *.jpg > out
identify: Corrupt JPEG data: premature end of data segment `145.jpg' @ warning/jpeg.c/JPEGWarningHandler/403.
identify: Corrupt JPEG data: 2 extraneous bytes before marker 0xd9 `146.jpg' @ warning/jpeg.c/JPEGWarningHandler/403.

The same corruption happens with Pillow 10.1.0 and 10.0.1 but does not happen with Pillow 10.0.0 on Mac ARM. The corruption does not happen in Intel Mac nor on Windows 11 with Pillow 10.1.0.

So it appears the bug is specific to Mac ARM Pillow 10.0.1 and 10.1.0. I see that the libjpeg*.dylib version changed in the Mac ARM Pillow in PIL/.dylibs in going from version 10.0.0 (no corruption, libjpeg.62.3.0.dylib) to version 10.0.1 and 10.1.0 (corruption with libjpeg.62.4.0.dylib).

[tomgoddard]

Here is the jpg_test.zip file.

jpg_test.zip

[radarhere]

Hi. Your description of the version involved, combined with the fact that I can't replicate this with a wheel built from main, makes me think this is a duplicate of #7477, #7604 and #7625.

Some users have found that libjpeg-turbo 3.0.0 causes some corruption. Our wheels only upgraded to that version with Pillow 10.0.1. The next release of Pillow, due out on January 2, will include libjpeg-turbo 3.0.1.

[tomgoddard]

I don't see any libjpeg-turbo library (.dylib) in the Pillow 10 Mac ARM PyPi distributions. It would be good to test a solution and not just wait and see. If a large fraction of the jpeg images written by Pillow since 10.0.1 (October 2023) on Mac ARM are slightly corrupted (sometimes visible in the lower right corner of images) this seems like a serious issue.

[tomgoddard]

In my test of 179 random thumbnail images 66 written by Pillow 10.1.0 were corrupted (more than 1/3). All the test images were 128 by 128 pixels in size. Not sure if the likelihood of corruption is related to the image dimensions.

[radarhere]

I was able to replicate the failure with a released wheel, but not with a main wheel.

I've now put together wheels from main with libjpeg-turbo 3.0.0 and libjpeg-turbo 3.0.1. Testing with Python 3.8, I find that 3.0.0 fails, but 3.0.1 works.

Here are wheels if you would like to test yourself.

pillow-10.2.0.dev0-cp311-cp311-macosx_11_0_arm64.whl-with300.zip
pillow-10.2.0.dev0-cp311-cp311-macosx_11_0_arm64.whl-with301.zip

[tomgoddard]

Thanks! Look forward to using the Jan 2 Pillow release.

[radarhere]

Pillow 10.2.0 has now been published, with libjpeg-turbo 3.0.1 in the wheels.

[tomgoddard]

I tested Pillow 10.2.0 on the jpg_test.zip test case and it has fixed the problem. Thanks!