libcrypt.so error with PyMuPDF-1.23.0

[chrismaille]

Please provide all mandatory information!

Describe the bug (mandatory)

Running poetry update got the warning. After that, when running code I get the following error:

    import fitz
E   ModuleNotFoundError: No module named 'fitz'

To Reproduce (mandatory)

• On pyproject.toml: PyMuPDF = "*"
• Run poetry update

Expected behavior (optional)

Dependencies must update correctly.

Screenshots (optional)

$ poetry update                                                                                                                                                                                                                                                                                    
Updating dependencies
Resolving dependencies... (11.5s)

Package operations: 2 installs, 3 updates, 0 removals

  • Updating pydantic-core (2.6.1 -> 2.6.3)
  • Installing pymupdfb (1.23.0)
  • Updating pydantic (2.2.1 -> 2.3.0)
  • Updating pymupdf (1.22.5 -> 1.23.0)
  • Installing pytest-asyncio (0.21.1)
Warning: Validation of the RECORD file of PyMuPDF-1.23.0-cp310-none-manylinux2014_x86_64.whl failed. Please report to the maintainers of that package so they can fix their build process. Details:
In /home/chris/.cache/pypoetry/artifacts/6a/f6/68/27aca004e5ad85ca6341bc534e5af81f2cf2b4de7348de2bc0c2fbf6a0/PyMuPDF-1.23.0-cp310-none-manylinux2014_x86_64.whl, PyMuPDF-1.23.0.dist-info/RECORD is not mentioned in RECORD

Your configuration (mandatory)

• Windows 11 WSL
• Python 3.10.12

[JorjMcKie]

In my Windows WSL (also Python 3.10), the vanilla installation procedure running pip works fine and leads to a usable fitz, i.e. python 3 -m pip install -U pymupdf does work.
Maybe you should do this before you update that other package.

[chrismaille]

I'm using Poetry to manage my project dependencies. Testing again using a new virtualenv:

$  poetry init                                                                                                                                                                                                                                                                                                         
This command will guide you through creating your pyproject.toml config.

Package name [pdf_test]:  
Version [0.1.0]:  
Description []:  
Author [XXX <XXX@users.noreply.github.com>, n to skip]:
License []:
Compatible Python versions [^3.10]:

Would you like to define your main dependencies interactively? (yes/no) [yes]
You can specify a package in the following forms:
  - A single name (requests): this will search for matches on PyPI
  - A name and a constraint (requests@^2.23.0)
  - A git url (git+https://github.com/python-poetry/poetry.git)
  - A git url with a revision (git+https://github.com/python-poetry/poetry.git#develop)
  - A file path (../my-package/my-package.whl)
  - A directory (../my-package/)
  - A url (https://example.com/packages/my-package-0.1.0.tar.gz)

Package to add or search for (leave blank to skip): pymupdf
Found 20 packages matching pymupdf
Showing the first 10 matches

Enter package # to add, or the complete package name if it is not listed []:
 [ 0] PyMuPDF
 [ 1] pymupdf-fonts
 [ 2] csv2img
 [ 3] PyMuPDFb
 [ 4] pdfautonup
 [ 5] vimura-server
 [ 6] pdf2pptx
 [ 7] odoo14-addon-account-invoice-import-simple-pdf
 [ 8] pyPDFeditor-GUI
 [ 9] TextSpitter
 [ 10]
 > 0
Enter the version constraint to require (or leave blank to use the latest version):
Using version ^1.23.0 for PyMuPDF

Add a package (leave blank to skip):

Would you like to define your development dependencies interactively? (yes/no) [yes] no
Generated file

[tool.poetry]
name = "pdf-test"
version = "0.1.0"
description = ""
authors = ["XXX <XXX@users.noreply.github.com>"]
readme = "README.md"
packages = [{include = "pdf_test"}]

[tool.poetry.dependencies]
python = "^3.10"
PyMuPDF = "^1.23.0"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"


Do you confirm generation? (yes/no) [yes] yes

/home/XXX/pdf_test/pdf_test does not contain any element

$  poetry shell                                                                                                                                                                                                                                                                                                      
Spawning shell within /home/XXX/.cache/pypoetry/virtualenvs/pdf-test-65aTmHUJ-py3.10
$  emulate bash -c '. /home/XXX/.cache/pypoetry/virtualenvs/pdf-test-65aTmHUJ-py3.10/bin/activate'                                                                                                                                                                                                        

$  python                                                                                                                                                                                                                                                                                                 
Python 3.10.12 (main, Jul  4 2023, 16:18:56) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import fitz
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/chris/.cache/pypoetry/virtualenvs/pdf-test-65aTmHUJ-py3.10/lib/python3.10/site-packages/fitz/__init__.py", line 22, in <module>
    from fitz.fitz import *
  File "/home/chris/.cache/pypoetry/virtualenvs/pdf-test-65aTmHUJ-py3.10/lib/python3.10/site-packages/fitz/fitz.py", line 14, in <module>
    from . import _fitz
ImportError: libcrypt.so.2: cannot open shared object file: No such file or directory

Reverting to 1.22.5 resolves the issue.

[JorjMcKie]

Ah, this is due to the slightly changed internal making of pymupdf ... and an internal bug of Python 3.10 on Linux.
File libcrypt.so.2 is listed as a requirement although unused. Please look it up - I forgot the details.
Anyway, you should have a libcrypt.so.1 (or libcrypt.so) somewhere. Copy that and name the target libcrypt.so.2, and all is fine.
This problem does not occurs in later Python versions, nor on Windows at all.

[chrismaille]

I don't want to be rude, but I don't think manually editing my dependencies is the best solution. The README states this library can be used "from Python 3.8 and up" - so, maybe it's not better to handle this "internal bug of Python 3.10 on linux", fixing the build process as described in the Warning above?

If this is not possible, please let me know if this project can be used on Python 3.10 on Linux safety.

[julian-smith-artifex-com]

We don't yet have a good solution to the libcrypt.so error. As mentioned above, it's actually a bug in the Python installation, so is not something we can fix in PyMuPDF itself.

But i'm fairly sure that the diagnostic from Poetry about the RECORD file is just a warning (which we'll fix in a future release) and is not related to the libcrypt.so error.

[KastanDay]

Duplicate of #2617

What's a short-term solution?

🎉 Update: Reverting to 1.22.5 resolves the issue (as above)

[julian-smith-artifex-com]

We have a possible fix for the libcrypt error.

I've uploaded a test PyMuPDF-1.23.1 wheel for Linux and Python-3.10, to test.pypi.org.

The new wheel can be tested with:

pip uninstall --yes pymupdf
pip install -i https://test.pypi.org/simple pymupdf

Please report back on whether this fixes the problem on Linux+Python-3.10.

[This wheel may also fix the poetry RECORD file warning.]

[chrismaille]

Test installing PyMUPDF 1.23.1 from test.pypi.org (Python 3.10.12 from Windows WSL):

$  poetry update                                                                                                                                                                                                                                                                                           
Updating dependencies
Resolving dependencies... (9.5s)

Package operations: 1 install, 0 updates, 0 removals

  • Installing pymupdf (1.23.1): Failed

  InvalidRecordEntry

  Row Index 19: expected 3 elements, got 1

  at ~/.pyenv/versions/3.10.12/lib/python3.10/site-packages/installer/records.py:214 in parse_record_file
      210│         if len(elements) != 3:
      211│             message = "Row Index {}: expected 3 elements, got {}".format(
      212│                 row_index, len(elements)
      213│             )
    → 214│             raise InvalidRecordEntry(elements=elements, issues=[message])
      215│
      216│         # Convert Windows paths to use / for consistency
      217│         elements[0] = elements[0].replace("\\", "/")
      218│

Warning: Validation of the RECORD file of PyMuPDF-1.23.1-cp310-none-manylinux2014_x86_64.whl failed. Please report to the maintainers of that package so they can fix their build process. Details:
Unable to retrieve `RECORD` from /home/chris/.cache/pypoetry/artifacts/22/b6/93/00433f6c217e523169a9655bdf8622dd2dd4fbc7e9209bd57d12efe406/PyMuPDF-1.23.1-cp310-none-manylinux2014_x86_64.whl: InvalidRecordEntry(elements=['PyMuPDF-1.23.1.dist-info/RECORD'], issues=['Row Index 19: expected 3 elements, got 1'])

[julian-smith-artifex-com]

Thanks for this. I have a further fix for the RECORD file issue, but there is no wheel available with this fix yet.

In the meantime, would you be able to try the suggested pip commands to test the new wheel fixes the underlying libcrypt.so problem?

pip uninstall --yes pymupdf
pip install -i https://test.pypi.org/simple pymupdf

[julian-smith-artifex-com]

We've just made a new release, PyMuPDF-1.23.1, which contains the fixes discussed earlier.

I've removed the test wheels from test.pypi.org.

[julian-smith-artifex-com]

Hopefully the new release of PyMuPDF-1.23.1 will fix this libcrypt.so issue, it certainly does in our internal testing.

Please let us know here if the new release fixes things for you also.

[The new release should also fix the Poetry warning about the RECORD file.]

[chrismaille]

PyMuPDF-1.23.1 working as intended. Thank you very much.

$ poetry install
Creating virtualenv pdf-test-ddD2c1aU-py3.10 in /home/chris/.cache/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies... (1.1s)

Package operations: 2 installs, 0 updates, 0 removals

  • Installing pymupdfb (1.23.0)
  • Installing pymupdf (1.23.1)

Writing lock file

/home/XX/Projects/pdf_test/pdf_test does not contain any element
$  poetry shell
Spawning shell within /home/XX/.cache/pypoetry/virtualenvs/pdf-test-ddD2c1aU-py3.10
$  emulate bash -c '. /home/XX/.cache/pypoetry/virtualenvs/pdf-test-ddD2c1aU-py3.10/bin/activate'
$  python
Python 3.10.12 (main, Jul  4 2023, 16:18:56) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import fitz
>>> fitz
<module 'fitz' from '/home/XX/.cache/pypoetry/virtualenvs/pdf-test-ddD2c1aU-py3.10/lib/python3.10/site-packages/fitz/__init__.py'>
>>>

[julian-smith-artifex-com]

That's great, thanks for confirming.