【v5.4】partially initialized module 'yaml' has no attribute '__with_libyaml__'

[sdn0303]

Hi there, Thank you for releasing the new version 5.4.
When I imported version 5.4, I found the following error and would like to report it.

2021-01-20 12:23:20 import yaml
2021-01-20 12:23:20 File "/root/.local/lib/python3.8/site-packages/yaml/__init__.py", line 13, in <module>
2021-01-20 12:23:20 from .cyaml import *
2021-01-20 12:23:20 File "/root/.local/lib/python3.8/site-packages/yaml/cyaml.py", line 7, in <module>
2021-01-20 12:23:20 from _yaml import CParser, CEmitter
2021-01-20 12:23:20 File "/usr/local/lib/python3.8/site-packages/_yaml/__init__.py", line 8, in <module>
2021-01-20 12:23:20 if not yaml.__with_libyaml__:
2021-01-20 12:23:20 AttributeError: partially initialized module 'yaml' has no attribute '__with_libyaml__' (most likely due to a circular import)

The reproduced environment is python: 3.8-slim docker image.

[p-riebs]

Seeing the same issue when using docker compose:

Traceback (most recent call last):
  File "/usr/local/bin/docker-compose", line 6, in <module>
    from compose.cli.main import main
  File "/usr/local/lib/python3.7/dist-packages/compose/cli/main.py", line 20, in <module>
    from ..config import ConfigurationError
  File "/usr/local/lib/python3.7/dist-packages/compose/config/__init__.py", line 3, in <module>
    from .config import ConfigurationError
  File "/usr/local/lib/python3.7/dist-packages/compose/config/config.py", line 12, in <module>
    import yaml
  File "/home/pi/.local/lib/python3.7/site-packages/yaml/__init__.py", line 13, in <module>
    from .cyaml import *
  File "/home/pi/.local/lib/python3.7/site-packages/yaml/cyaml.py", line 7, in <module>
    from _yaml import CParser, CEmitter
  File "/usr/local/lib/python3.7/dist-packages/_yaml/__init__.py", line 8, in <module>
    if not yaml.__with_libyaml__:
AttributeError: module 'yaml' has no attribute '__with_libyaml__'

[florimondmanca]

Hi,

I'm seeing the same issue in some CI builds that run on Linux (GitHub Actions). FWIW, I'm not able to reproduce on my macOS machine (running Python 3.8.5):

$ python
Python 3.8.5 (default, Aug 13 2020, 15:19:10) 
[Clang 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import yaml
>>> yaml.__version__
'5.4'

For anyone passing by, a workaround for now would be to pin to pyyaml<5.4, I suppose?

Is there anything I can do to help resolve this?

[RichaSharmaTR]

upgrading pyyaml version in requirements.txt file fixed this issue for me.
PyYAML==5.4

[thanos1983]

upgrading pyyaml version in requirements.txt file fixed this issue for me.
PyYAML==5.4

For me actually going from PyYAML==5.2.1 to 5.4 broke my ansible.

The solution was to revert to 5.3.

Sample of problem:

$ ansible --version
Traceback (most recent call last):
  File "/usr/local/bin/ansible", line 62, in <module>
    import ansible.constants as C
  File "/home/<user>/.local/lib/python3.8/site-packages/ansible/constants.py", line 15, in <module>
    from ansible.config.manager import ConfigManager, ensure_type, get_ini_config_value
  File "/home/<user>/.local/lib/python3.8/site-packages/ansible/config/manager.py", line 17, in <module>
    from yaml import load as yaml_load
  File "/home/<user>/.local/lib/python3.8/site-packages/yaml/__init__.py", line 13, in <module>
    from .cyaml import *
  File "/home/<user>/.local/lib/python3.8/site-packages/yaml/cyaml.py", line 7, in <module>
    from _yaml import CParser, CEmitter
  File "/usr/local/lib/python3.8/dist-packages/_yaml/__init__.py", line 8, in <module>
    if not yaml.__with_libyaml__:
AttributeError: partially initialized module 'yaml' has no attribute '__with_libyaml__' (most likely due to a circular import)
$ sudo -H python3 -m pip install PyYAML==5.3
Collecting PyYAML==5.3
Installing collected packages: PyYAML
  Found existing installation: PyYAML 5.4
    Uninstalling PyYAML-5.4:
      Successfully uninstalled PyYAML-5.4
Successfully installed PyYAML-5.3
$ python3 -V
Python 3.8.7
$ ansible --version
ansible 2.10.4
  config file = /home/<user>/cookbook/ansible.cfg
  configured module search path = ['/home/<user>/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/<user>/.local/lib/python3.8/site-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.8.7 (default, Dec 21 2020, 20:10:35) [GCC 7.5.0]

Hope this helps.

[laurentvm]

Same with python 3.8 and PyYAML 5.4.

Try to build with libyaml and install.

Get same error:

  File "/home/laurent/git/python/python3/lib/python3.8/site-packages/PyYAML-5.4-py3.8-linux-x86_64.egg/_yaml/__init__.py", line 8, in <module>                                                                  
    if not yaml.__with_libyaml__:                                                                                                                                                                              
AttributeError: partially initialized module 'yaml' has no attribute '__with_libyaml__' (most likely due to a circular import)

[bsolomon1124]

@sdn0303 , did you install with any pip options or is any info possibly excluded there?

Here is a python:3.8-slim shell:

$ docker container run -it --rm --entrypoint=bash python:3.8-slim
root@11a5089d68cf:/# python3 -m pip install 'PyYAML==5.4'
Collecting PyYAML==5.4
  Downloading PyYAML-5.4-cp38-cp38-manylinux1_x86_64.whl (662 kB)
     |████████████████████████████████| 662 kB 6.4 MB/s 
Installing collected packages: PyYAML
Successfully installed PyYAML-5.4
root@11a5089d68cf:/# python3 -c 'import yaml; print(yaml.__with_libyaml__)'
True

root@11a5089d68cf:/# python3 -m pip install 'ansible==2.9.*'
Collecting ansible==2.9.*
  Downloading ansible-2.9.17.tar.gz (14.3 MB)
     |████████████████████████████████| 14.3 MB 6.7 MB/s 
Requirement already satisfied: PyYAML in /usr/local/lib/python3.8/site-packages (from ansible==2.9.*) (5.4)

root@11a5089d68cf:/# ansible --version
ansible 2.9.17
  config file = None
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.8/site-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.8.7 (default, Jan 12 2021, 17:16:32) [GCC 8.3.0]

Ok also with Ansible 2.10.5.

[bsolomon1124]

Also having trouble reproducing with --no-binary, here again on python:3.8-slim:

root@3323cf48a2d3:/# apt-get update -yq
root@3323cf48a2d3:/# apt-get install -yq --no-install-recommends gcc libc6-dev linux-libc-dev
root@3323cf48a2d3:/# python3 -m pip install --no-binary=:all: PyYAML
root@3323cf48a2d3:/# python3 -c 'import yaml; print(yaml.__with_libyaml__)'
False

[sdn0303]

@bsolomon1124
Hi,

I use pipenv to manage my dependencies and install them with the 'pipenv install --system' command defined in the Dockerfile.

[nitzmahone]

@thanos1983 at a glance, it looks like your issue is a mixed install- it appears to be loading the _yaml package from dist-packages instead of site-packages. I'm guessing your venv is configured to allow fallback to system modules? It still shouldn't be an issue, but the change in dirs looks pretty suspicious.

[nitzmahone]

@florimondmanca can you poke around on the filesystem a bit for one of those failures? I'm curious if there's a _yaml package as a sibling to wherever you're loading yaml from (eg, whatever/lib64/python3.8/site-packages). The traceback @thanos1983 implies that there isn't on that system, so I'm curious if you're seeing the exact same thing.

[nitzmahone]

I guess the other possibility is that there's an old system/distutils-owned _yaml package on a higher level of the path. That sorta implies a misconfigured path, but we need to be ultra careful about letting that load, since that's also where the old C ext lives- we don't want to let it load twice.

[bsolomon1124]

Yes, I was wondering if this might be due to a prior/existing install of _yaml < 5.4 where _yaml is the actual extension module rather than the package created in 5.4.

@sdn0303, if you are able to share more specific detail to help reproduce what's happening here, that would help to get things solved all that much faster.

[nitzmahone]

Yeah, I'm pretty sure that's exactly what's happening, but unless it's a half uninstalled module or something, it's unclear how they're not getting yaml from the same location. This is (one of many) reasons why pip killed support for uninstalling distutils-owned packages, methinks... :(

[nitzmahone]

I'm going to try and contrive an install like that with just _yaml from 5.3 in a higher precedence PYTHONPATH entry. If that's the case, we might be able to detect it and at least fail in a more descriptive manner (eg, "you've got a half-baked pyyaml install at $path, go clean it up!")

[bsolomon1124]

Confirmed it does not seem to be a pipenv issue. (did not suspect it would be, but looks like that can be crossed off.)

[nitzmahone]

I think I know what's happening- it's a little more contrived than what we thought, but a variation on a theme (old yaml package built without C ext tries to load new _yaml stub package from a lower PYTHONPATH entry). If I'm right, I think we can tweak the _yaml stub package in 5.4+ to fail in a compatible way that will still leave old versions of pyyaml that try to load it functional.

[nitzmahone]

Yep, that's it- I have a local repro, and will play with a fix right now.

[nitzmahone]

The underlying failure here should only occur in a mixed version environment where an old version of pyyaml sees a new version of the stub _yaml package (so for you folks that are hitting this- are you sure you're using the pyyaml version you think you are?). Further, it would only occur when the older pyyaml version was built without the C extension, and it had visibility into another site-packages (et al) dir with the new version of pyyaml- the old pyyaml loads the new stub package, which tried to interrogate an attribute that doesn't exist on the old yaml module. The fix changes that interrogation to be more defensive and to simulate an ImportError in the case of a version mismatch.

[nitzmahone]

Release 5.4.1 in flight with the fix.

[florimondmanca]

@nitzmahone Thank you!

[bsolomon1124]

@florimondmanca it looks like PyYAML 5.4.1 is up on PyPI, are you able to confirm that AttributeError goes away with the patch version release?

[sdn0303]

@nitzmahone @bsolomon1124
The error has been resolved. Thank you！🙏

[thanos1983]

Release 5.4.1 in flight with the fix.

I can confirm the issue is resolved in the new release. Thanks @nitzmahone for the quick fix.

Sample of resolution:

$ sudo -H python3 -m pip install PyYAML -U
Collecting PyYAML
  Downloading https://repo/api/pypi/pypi-remote/packages/packages/70/96/c7245e551b1cb496bfb95840ace55ca60f20d3d8e33d70faf8c78a976899/PyYAML-5.4.1-cp38-cp38-manylinux1_x86_64.whl (662kB)
    100% |████████████████████████████████| 665kB 1.7MB/s
Installing collected packages: PyYAML
  Found existing installation: PyYAML 5.3
    Uninstalling PyYAML-5.3:
      Successfully uninstalled PyYAML-5.3
Successfully installed PyYAML-5.4.1
$ ansible --version
ansible 2.10.4
  config file = /home/<user>/cookbook/ansible.cfg
  configured module search path = ['/home/<user>/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/<user>/.local/lib/python3.8/site-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.8.7 (default, Dec 21 2020, 20:10:35) [GCC 7.5.0]