import xmlsec fails on python 3.8/3.9, macOS Big Sur 11.0.1 and M1 chip

[tesh-raichandani]

xmlsec version: 1.3.9
libxmlsec1 version: 1.2.31
libxml2 version: 2.9.10_2
(venv) tesh$ pkg-config --cflags xmlsec1
-D__XMLSEC_FUNCTION__=__func__ -DXMLSEC_NO_SIZE_T -DXMLSEC_NO_GOST=1 -DXMLSEC_NO_GOST2012=1 -DXMLSEC_NO_CRYPTO_DYNAMIC_LOADING=1 -DXMLSEC_CRYPTO_OPENSSL=1 -I/opt/homebrew/Cellar/libxmlsec1/1.2.31/include/xmlsec1 -I/opt/homebrew/opt/openssl@1.1/include -I/opt/homebrew/opt/openssl@1.1/include/openssl

It installs fine but fails when importing.

In [1]: import xmlsec
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-43dd76db73dd> in <module>
----> 1 import xmlsec

ImportError: dlopen(/Users/tesh/rescale/venv3.9/lib/python3.9/site-packages/xmlsec.cpython-39-darwin.so, 2): Symbol not found: _xmlSecDSigNs
  Referenced from: /Users/tesh/rescale/venv3.9/lib/python3.9/site-packages/xmlsec.cpython-39-darwin.so
  Expected in: flat namespace
 in /Users/tesh/rescale/venv3.9/lib/python3.9/site-packages/xmlsec.cpython-39-darwin.so

Python 3.8 and 3.9

$ file $(which python)
/Users/tesh/rescale/venv/bin/python: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64] [arm64:Mach-O 64-bit executable arm64]
/Users/tesh/rescale/venv/bin/python (for architecture x86_64):	Mach-O 64-bit executable x86_64
/Users/tesh/rescale/venv/bin/python (for architecture arm64):	Mach-O 64-bit executable arm64

$ file $(which python)
/Users/tesh/rescale/venv3.9/bin/python: Mach-O 64-bit executable x86_64

$ otool -L /Users/tesh/rescale/venv3.9/lib/python3.9/site-packages/xmlsec.cpython-39-darwin.so
/Users/tesh/rescale/venv3.9/lib/python3.9/site-packages/xmlsec.cpython-39-darwin.so:
	/usr/lib/libxslt.1.dylib (compatibility version 3.0.0, current version 3.26.0)
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.60.1)
	/usr/lib/libicucore.A.dylib (compatibility version 1.0.0, current version 66.1.0)

Will really appreciate any help.

[hoefling]

Hi @hitesh-raichandani, it looks like you have a Mac with the new M1 chip. Unfortunately, I don't have access to this hardware yet and Github Actions doesn't offer anything either, see actions/runner-images#2187. I'm thus unable to play with building universal dists right now. Once my company sets up the infrastructure, I will come back to this, or maybe someone else will shed some light on it.

[tesh-raichandani]

Thanks for the update @hoefling, added package versions to the issue details for clarity.

[hoefling]

@hitesh-raichandani I still can't reproduce the issue due to missing hardware, but maybe you could attach the build log in the meantime? Simply pip install xmlsec -vvv --force-reinstall --no-cache-dir and drop the full output in the comments.

[tesh-raichandani]

@hoefling, the attached file has build logs. Thanks!

xmlsec-build-log.txt

[px0]

I'm running into the same issue except I am running python3.7 in the Rosetta2 compatibility mode

[milannister]

Same issue here with python 3.8 and M1.

[aldenjenkins]

Same issue found here. Any fix yet?

[hoefling]

I can't investigate any issues with M1 since I don't own the necessary hardware. A colleague of mine could successfully compile xmlsec from current master though on his MacBook Pro M1, using the brewed libraries. The attached xmlsec-apple-m1.zip contains both wheels for xmlsec and lxml, which you can try out.

Beware though, that lxml seems to continue having issues with the import order, resulting in XMLSyntaxErrors if reusing the default parser instance; see related #84, #96 and the issues in lxml's tracker: 1880251, 1887848. Failing example:

import pathlib
pathlib.Path("doc.xml").write_text("<spam>eggs</spam>")

import xmlsec
from lxml import etree

etree.parse("doc.xml")
etree.parse("doc.xml")

[Gyanesha]

I too have found this very same issue, any fix for this ?

[andribergs]

I get the same issue with python 3.9 on a M1 chip, any updates on this?

[milannister]

Hi there, are there any plans on progressing on this issue anytime soon? I understand the constraints, and would just like to get some feedback so that we can plan for alternatives if needed.

[mrmiguu]

Howdy! Any updates on the progression for this issue? I understand the scenario and would like to hear back regarding workarounds.

Thanks

[spacewaffle]

Still having this issue on python 3.9 and M1. Any updates?

[hoefling]

There's nothing to report. The company I worked at didn't manage to provide the M1 hardware and I changed the job recently, so no ETA at all. Feel free to give a 👍 on actions/runner-images#2187, as accessing the M1 hardware via CI jobs will be most likely the only way for me.

[armenzg]

It installed on my M1 machine.
You will need to build the wheel, thus, you will need to install libxmlsec1 and pkg-config.
You might need to brew update and reinstall those packages.
I have been able to builds the wheels for these versions: 1.3.10 to 1.3.12.

Anyone trying this issue also make sure to only try with Python versions 3.8.10 and greater since that's when official support was added.

❯ pip install xmlsec==1.3.12 --no-cache-dir
Collecting xmlsec==1.3.12
  Downloading xmlsec-1.3.12.tar.gz (64 kB)
     |████████████████████████████████| 64 kB 3.0 MB/s
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
    Preparing wheel metadata ... done
Requirement already satisfied: lxml>=3.8 in ./.venv/lib/python3.8/site-packages (from xmlsec==1.3.12) (4.6.3)
Building wheels for collected packages: xmlsec
  Building wheel for xmlsec (PEP 517) ... done
  Created wheel for xmlsec: filename=xmlsec-1.3.12-cp38-cp38-macosx_11_0_arm64.whl size=48084 sha256=74d628b8edf4a81bde35e4a0863e2c1d2e83a7f897c0c555c1612e6b25fb1ad4
  Stored in directory: /private/var/folders/j1/d109sd4n55qbk5ty8gzq5jgm0000gn/T/pip-ephem-wheel-cache-rt5nmgzu/wheels/2c/99/a3/98bcb24586c778eb35e23e30af30fbf1c34a0a9ed85e18576a
Successfully built xmlsec
Installing collected packages: xmlsec
Successfully installed xmlsec-1.3.12

❯ pkg-config --cflags xmlsec1
-D__XMLSEC_FUNCTION__=__func__ -DXMLSEC_NO_SIZE_T -DXMLSEC_NO_GOST=1 -DXMLSEC_NO_GOST2012=1 -DXMLSEC_NO_CRYPTO_DYNAMIC_LOADING=1 -DXMLSEC_CRYPTO_OPENSSL=1 -I/opt/homebrew/Cellar/libxmlsec1/1.2.32/include/xmlsec1 -I/opt/homebrew/opt/openssl@1.1/include -I/opt/homebrew/opt/openssl@1.1/include/openssl

❯ python -c "import xmlsec"

❯ file $(which python)
/Users/armenzg/code/sentry/.venv/bin/python: Mach-O 64-bit executable arm64

❯ otool -L .venv/lib/python3.8/site-packages/xmlsec.cpython-38-darwin.so
.venv/lib/python3.8/site-packages/xmlsec.cpython-38-darwin.so:
	/opt/homebrew/opt/libxmlsec1/lib/libxmlsec1-openssl.1.dylib (compatibility version 4.0.0, current version 4.32.0)
	/opt/homebrew/opt/libxmlsec1/lib/libxmlsec1.1.dylib (compatibility version 4.0.0, current version 4.32.0)
	/opt/homebrew/opt/openssl@1.1/lib/libcrypto.1.1.dylib (compatibility version 1.1.0, current version 1.1.0)
	/usr/lib/libxslt.1.dylib (compatibility version 3.0.0, current version 3.26.0)
	/usr/lib/libxml2.2.dylib (compatibility version 10.0.0, current version 10.9.0)
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.100.5)
	/usr/lib/libicucore.A.dylib (compatibility version 1.0.0, current version 66.1.0)

❯ brew info libxmlsec1
libxmlsec1: stable 1.2.32 (bottled)
XML security library
https://www.aleksey.com/xmlsec/
/opt/homebrew/Cellar/libxmlsec1/1.2.32 (220 files, 6.2MB) *
  Poured from bottle on 2021-05-17 at 08:58:46
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/libxmlsec1.rb
License: MIT
==> Dependencies
Build: pkg-config ✔
Required: gnutls ✔, libgcrypt ✘, libxml2 ✔, openssl@1.1 ✔
==> Requirements
Build: Xcode ✘

❯ brew info pkg-config
pkg-config: stable 0.29.2 (bottled)
Manage compile and link flags for libraries
https://freedesktop.org/wiki/Software/pkg-config/
/opt/homebrew/Cellar/pkg-config/0.29.2_3 (11 files, 677.1KB) *
  Poured from bottle on 2021-05-17 at 08:53:19
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/pkg-config.rb
License: GPL-2.0-or-later