`av==9.1.1` wheels seem to be be not packaged properly for Windows and Python 3.8/9

[pmeier]

Overview

av==9.1.1 wheel released a couple of hours ago seems to be not packaged properly for Windows and Python 3.8 and 3.9.

Expected behavior

Import without issues.

Actual behavior

Traceback (most recent call last):
  File "C:\Users\circleci\project\test\datasets_utils.py", line 90, in _import
    module = importlib.import_module(package)
  File "C:\Users\circleci\project\env\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "C:\Users\circleci\project\env\lib\site-packages\av\__init__.py", line 44, in <module>
    from av._core import time_base, library_versions
ImportError: DLL load failed while importing _core: The specified module could not be found.

Investigation

CI runs:

• ✔️ Python 3.7
• ✖️ Python 3.8
• ✖️ Python 3.9
• ✔️ Python 3.10

Reproduction

import av

Versions

• OS: Windows
• PyAV runtime: 9.1.1

[pmeier]

For reference, here are the two CI runs after we pinned av != 9.1.1:

• ✔️ Python 3.8
• ✔️ Python 3.9

[jlaine]

There was a significant change in this release which is that the wheels bow use an FFmpeg built from source (same as the other platforms), sorry for the breakage.

What intrigues me:

• is that the failure is Python-version-specific
• that we didn't get a CI failure

It sounds like there is a DLL missing, or clashing with another.. as I don't have access to a Windows machine this is going to be tricky to investigate.

[jlaine]

I just checked and CI definitely tested the 3.8 and 3.9 wheels on Windows:

https://github.com/PyAV-Org/PyAV/runs/5783121554?check_suite_focus=true

This leaves me wondering whether the wheels are actually broken or if we have a conflict with pytorch?

[pmeier]

is that the failure is Python-version-specific

Not only version specific, but also the "middle" ones. 3.7 and 3.10 are fine, which is especially peculiar.

This leaves me wondering whether the wheels are actually broken or if we have a conflict with pytorch?

I'll try to investigate on our side. @bjuncek are you aware of anything on our side that could cause this?

[pmeier]

Error persists even if we only setup the environment and not install anything form the PyTorch ecosystem. Thus, if there is a DLL conflict, it is not coming from us. Here are the envs:

3.7

# Name                    Version                   Build  Channel
atomicwrites              1.4.0                      py_0  
attrs                     21.4.0             pyhd3eb1b0_0  
av                        9.1.1                    pypi_0    pypi
ca-certificates           2022.3.29            haa95532_0  
cached-property           1.5.2                    pypi_0    pypi
certifi                   2021.10.8        py37haa95532_2  
colorama                  0.4.4              pyhd3eb1b0_0  
coverage                  6.3.2            py37h2bbff1b_0  
dataclasses               0.6                      pypi_0    pypi
future                    0.18.2                   pypi_0    pypi
h5py                      3.6.0                    pypi_0    pypi
hdf5                      1.12.0               h1756f20_0  
icc_rt                    2019.0.0             h0cc432a_1  
importlib-metadata        4.11.3           py37haa95532_0  
importlib_metadata        4.11.3               hd3eb1b0_0  
iniconfig                 1.1.1              pyhd3eb1b0_0  
jpeg                      9d                   h2bbff1b_0  
libpng                    1.6.37               h2a8f88b_0  
numpy                     1.21.5                   pypi_0    pypi
openssl                   1.1.1n               h2bbff1b_0  
packaging                 21.3               pyhd3eb1b0_0  
pillow                    9.1.0                    pypi_0    pypi
pip                       21.2.4           py37haa95532_0  
pluggy                    1.0.0            py37haa95532_1  
py                        1.11.0             pyhd3eb1b0_0  
pyparsing                 3.0.4              pyhd3eb1b0_0  
pytest                    6.2.5            py37haa95532_2  
pytest-cov                3.0.0              pyhd3eb1b0_0  
pytest-mock               3.6.1              pyhd3eb1b0_0  
python                    3.7.13               h6244533_0  
scipy                     1.7.3                    pypi_0    pypi
setuptools                58.0.4           py37haa95532_0  
sqlite                    3.38.2               h2bbff1b_0  
toml                      0.10.2             pyhd3eb1b0_0  
typing_extensions         4.1.1              pyh06a4308_0  
vc                        14.2                 h21ff451_1  
vs2015_runtime            14.27.29016          h5e58377_2  
wheel                     0.37.1             pyhd3eb1b0_0  
wincertstore              0.2              py37haa95532_2  
zipp                      3.7.0              pyhd3eb1b0_0  
zlib                      1.2.11               hbd8134f_5

3.8

# Name                    Version                   Build  Channel
atomicwrites              1.4.0                      py_0  
attrs                     21.4.0             pyhd3eb1b0_0  
av                        9.1.1                    pypi_0    pypi
ca-certificates           2022.3.29            haa95532_0  
certifi                   2021.10.8        py38haa95532_2  
colorama                  0.4.4              pyhd3eb1b0_0  
coverage                  6.3.2            py38h2bbff1b_0  
dataclasses               0.6                      pypi_0    pypi
future                    0.18.2                   pypi_0    pypi
h5py                      3.6.0                    pypi_0    pypi
hdf5                      1.12.0               h1756f20_0  
icc_rt                    2019.0.0             h0cc432a_1  
iniconfig                 1.1.1              pyhd3eb1b0_0  
jpeg                      9d                   h2bbff1b_0  
libpng                    1.6.37               h2a8f88b_0  
numpy                     1.22.3                   pypi_0    pypi
openssl                   1.1.1n               h2bbff1b_0  
packaging                 21.3               pyhd3eb1b0_0  
pillow                    9.1.0                    pypi_0    pypi
pip                       21.2.2           py38haa95532_0  
pluggy                    1.0.0            py38haa95532_1  
py                        1.11.0             pyhd3eb1b0_0  
pyparsing                 3.0.4              pyhd3eb1b0_0  
pytest                    6.2.5            py38haa95532_2  
pytest-cov                3.0.0              pyhd3eb1b0_0  
pytest-mock               3.6.1              pyhd3eb1b0_0  
python                    3.8.13               h6244533_0  
scipy                     1.8.0                    pypi_0    pypi
setuptools                58.0.4           py38haa95532_0  
sqlite                    3.38.2               h2bbff1b_0  
toml                      0.10.2             pyhd3eb1b0_0  
vc                        14.2                 h21ff451_1  
vs2015_runtime            14.27.29016          h5e58377_2  
wheel                     0.37.1             pyhd3eb1b0_0  
wincertstore              0.2              py38haa95532_2  
zlib                      1.2.11               hbd8134f_5

3.9

# Name                    Version                   Build  Channel
atomicwrites              1.4.0                      py_0  
attrs                     21.4.0             pyhd3eb1b0_0  
av                        9.1.1                    pypi_0    pypi
ca-certificates           2022.3.29            haa95532_0  
certifi                   2021.10.8        py39haa95532_2  
colorama                  0.4.4              pyhd3eb1b0_0  
coverage                  6.3.2            py39h2bbff1b_0  
dataclasses               0.6                      pypi_0    pypi
future                    0.18.2                   pypi_0    pypi
h5py                      3.6.0                    pypi_0    pypi
hdf5                      1.12.0               h1756f20_0  
icc_rt                    2019.0.0             h0cc432a_1  
iniconfig                 1.1.1              pyhd3eb1b0_0  
jpeg                      9d                   h2bbff1b_0  
libpng                    1.6.37               h2a8f88b_0  
numpy                     1.22.3                   pypi_0    pypi
openssl                   1.1.1n               h2bbff1b_0  
packaging                 21.3               pyhd3eb1b0_0  
pillow                    9.1.0                    pypi_0    pypi
pip                       21.2.4           py39haa95532_0  
pluggy                    1.0.0            py39haa95532_1  
py                        1.11.0             pyhd3eb1b0_0  
pyparsing                 3.0.4              pyhd3eb1b0_0  
pytest                    6.2.5            py39haa95532_2  
pytest-cov                3.0.0              pyhd3eb1b0_0  
pytest-mock               3.6.1              pyhd3eb1b0_0  
python                    3.9.11               h6244533_2  
scipy                     1.8.0                    pypi_0    pypi
setuptools                58.0.4           py39haa95532_0  
sqlite                    3.38.2               h2bbff1b_0  
toml                      0.10.2             pyhd3eb1b0_0  
tzdata                    2022a                hda174b7_0  
vc                        14.2                 h21ff451_1  
vs2015_runtime            14.27.29016          h5e58377_2  
wheel                     0.37.1             pyhd3eb1b0_0  
wincertstore              0.2              py39haa95532_2  
zlib                      1.2.11               hbd8134f_5

3.10

# Name                    Version                   Build  Channel
atomicwrites              1.4.0                      py_0  
attrs                     21.4.0             pyhd3eb1b0_0  
av                        9.1.1                    pypi_0    pypi
bzip2                     1.0.8                he774522_0  
ca-certificates           2022.3.29            haa95532_0  
certifi                   2021.5.30       py310haa95532_0  
colorama                  0.4.4              pyhd3eb1b0_0  
coverage                  6.3.2           py310h2bbff1b_0  
dataclasses               0.6                      pypi_0    pypi
future                    0.18.2                   pypi_0    pypi
h5py                      3.6.0                    pypi_0    pypi
hdf5                      1.12.0               h1756f20_0  
icc_rt                    2019.0.0             h0cc432a_1  
iniconfig                 1.1.1              pyhd3eb1b0_0  
jpeg                      9d                   h2bbff1b_0  
libffi                    3.4.2                h604cdb4_1  
libpng                    1.6.37               h2a8f88b_0  
more-itertools            8.12.0             pyhd3eb1b0_0  
numpy                     1.22.3                   pypi_0    pypi
openssl                   1.1.1n               h2bbff1b_0  
packaging                 21.3               pyhd3eb1b0_0  
pillow                    9.1.0                    pypi_0    pypi
pip                       21.2.4          py310haa95532_0  
pluggy                    0.13.1          py310haa95532_0  
py                        1.11.0             pyhd3eb1b0_0  
pyparsing                 3.0.4              pyhd3eb1b0_0  
pytest                    6.2.4           py310haa95532_2  
pytest-cov                3.0.0              pyhd3eb1b0_0  
pytest-mock               3.6.1              pyhd3eb1b0_0  
python                    3.10.4               hbb2ffb3_0  
scipy                     1.8.0                    pypi_0    pypi
setuptools                58.0.4          py310haa95532_0  
sqlite                    3.38.2               h2bbff1b_0  
tk                        8.6.11               h2bbff1b_0  
toml                      0.10.2             pyhd3eb1b0_0  
tzdata                    2022a                hda174b7_0  
vc                        14.2                 h21ff451_1  
vs2015_runtime            14.27.29016          h5e58377_2  
wheel                     0.37.1             pyhd3eb1b0_0  
wincertstore              0.2             py310haa95532_2  
xz                        5.2.5                h62dcd97_0  
zlib                      1.2.11               hbd8134f_5

Is there a "verbose mode" or the like to find which DLL breaks the import?

[pmeier]

Failure also happens if nothing but pyav is installed:

3.8

# Name                    Version                   Build  Channel
av                        9.1.1                    pypi_0    pypi
ca-certificates           2022.3.29            haa95532_0  
certifi                   2021.10.8        py38haa95532_2  
openssl                   1.1.1n               h2bbff1b_0  
pip                       21.2.2           py38haa95532_0  
python                    3.8.13               h6244533_0  
setuptools                58.0.4           py38haa95532_0  
sqlite                    3.38.2               h2bbff1b_0  
vc                        14.2                 h21ff451_1  
vs2015_runtime            14.27.29016          h5e58377_2  
wheel                     0.37.1             pyhd3eb1b0_0  
wincertstore              0.2              py38haa95532_2

3.9

# Name                    Version                   Build  Channel
av                        9.1.1                    pypi_0    pypi
ca-certificates           2022.3.29            haa95532_0  
certifi                   2021.10.8        py39haa95532_2  
openssl                   1.1.1n               h2bbff1b_0  
pip                       21.2.4           py39haa95532_0  
python                    3.9.11               h6244533_2  
setuptools                58.0.4           py39haa95532_0  
sqlite                    3.38.2               h2bbff1b_0  
tzdata                    2022a                hda174b7_0  
vc                        14.2                 h21ff451_1  
vs2015_runtime            14.27.29016          h5e58377_2  
wheel                     0.37.1             pyhd3eb1b0_0  
wincertstore              0.2              py39haa95532_2

[jlaine]

I had a closer look at the DLLs contained in the av 9.1.0 vs 9.1.1 wheels and:

• the DLL names seem to be properly mangled so we shouldn't experience "DLL hell"
• the only additional required library seems to be MFPlat.DLL (media foundation), which I understand may not be present on all Windows installations.

If I provide you with a wheel for a Python version of your choice would you be able to test it? Otherwise it means going through a full release process, potentially with no gain..

[pmeier]

Yes, we could test if we had a wheel available.

[jlaine]

OK, the FFmpeg rebuild has completed (11 hours to cover all platforms..), and I've put together PR #954 to produce updated wheels. I'll keep you posted.

[jlaine]

Here are the wheels for Windows/64bit.. As github won't allow me to upload .whl files, nor to put them all in one big zip, I've had to put each in its .zip, sorry for the inconvenience!

av-9.1.1-cp37-cp37m-win_amd64.whl.zip
av-9.1.1-cp38-cp38-win_amd64.whl.zip
av-9.1.1-cp39-cp39-win_amd64.whl.zip
av-9.1.1-cp310-cp310-win_amd64.whl.zip