Unknown Python implementation: pyston (cannot install SciPy >= 1.9.0 on Pyston)

[PLPeeters]

This is similar to #121, only it's for the lesser-known (but awesome) Pyston.

NotImplementedError: Unknown Python implementation: pyston. Please report this to https://github.com/FFY00/mesonpy/issues and include information about the Python distribution you are using.

Could you add support for it?

[FFY00]

We can look into it, do you have some wheels that you can share? I need to see the values that are expected for Pyston.

[PLPeeters]

Sure, here's a ZIP file with two wheels built for Pyston: wheels.zip

[rgommers]

Thanks @PLPeeters. To make this easier for readers, here are the details. Wheel file names:

adns-1.4_py0-pyston38-pyston_23_x86_64_linux_gnu-linux_x86_64.whl
cffi-1.15.1-pyston38-pyston_23_x86_64_linux_gnu-linux_x86_64.whl

Content of the first wheel:

├── adns-1.4_py0.dist-info
│   ├── METADATA
│   ├── RECORD
│   ├── top_level.txt
│   └── WHEEL
├── ADNS.py
├── adns.pyston-23-x86_64-linux-gnu.so
└── DNSBL.py

Content of the second wheel:

.
├── cffi
│   ├── api.py
│   ├── backend_ctypes.py
│   ├── _cffi_errors.h
│   ├── _cffi_include.h
│   ├── cffi_opcode.py
│   ├── commontypes.py
│   ├── cparser.py
│   ├── _embedding.h
│   ├── error.py
│   ├── ffiplatform.py
│   ├── __init__.py
│   ├── lock.py
│   ├── model.py
│   ├── parse_c_type.h
│   ├── pkgconfig.py
│   ├── recompiler.py
│   ├── setuptools_ext.py
│   ├── vengine_cpy.py
│   ├── vengine_gen.py
│   └── verifier.py
├── cffi-1.15.1.dist-info
│   ├── entry_points.txt
│   ├── LICENSE
│   ├── METADATA
│   ├── RECORD
│   ├── top_level.txt
│   └── WHEEL
└── _cffi_backend.pyston-23-x86_64-linux-gnu.so

I'm not sure that these wheels are actually correct. The pyston-23 part looks tricky. PyPy doesn't have a hyphen there, it uses tags like pypy38, pypy39.

[FFY00]

I'm not sure that these wheels are actually correct.

To me, it looks reasonable. These values were probably first defined by distutils/setuptools, and it makes sense that for unknown implementations, it uses the extension name as the ABI tag -- we could probably do the same in the future, but for now we have enough issues 😅

Hey @kmod, could you clarify if these wheel tags are correct? Is there any other guidance you may want to provide regarding Pyston wheels? Thanks!

I'm not sure that these wheels are actually correct. The pyston-23 part looks tricky. PyPy doesn't have a hyphen there, it uses tags like pypy38, pypy39.

The 2.3 is the Pyston version, PyPy also includes it in the tags, just in a slightly different format.

My takeaways from these wheels are:

• The Python tag is pyston{sysconfig.get_config_var('py_version_nodot')}
• The ABI tag is ext_name.lower()

[kmod]

If I remember correctly the abi tag situation was changed recently (less than a year ago). We requested a 2-letter ABI tag, since this was previously required, but the python packaging team decided to move away from the 2-letter tag scheme. I forget the exact details of what they changed, and I was unable to find the thread where this happened.

@undingen do you remember more about this?

[undingen]

I'm not following what this thread is about but maybe this information about the Pyston PEP425 tag is helpful:

This was the Thread where we tried to get out own tag but ended up not getting/doing it: https://discuss.python.org/t/pep425-python-tag-for-pyston/13039
So if I remember correctly we did not end up changing the tag - just sticked to the default tag.

This was quite a long time ago so I can't really remember so I took a quick look how the wheel filename gets calculated:
The packaging package does all the work in this path packaging.tags.generic_tags():
https://github.com/pypa/packaging/blob/5f46d1532f45ae32cd7831c6f45212f7066c1ed0/packaging/tags.py#L254

The PEP425 tag comes down to:
<interpreter>-<abi>-<platform>
Where:

• interpreter is pyston38: sys.implementation.name + sysconfig.get_config_var("py_version_nodot")
• abi is pyston_23_x86_64_linux_gnu: sysconfig.get_config_var("SOABI").replace(".", "_").replace("-", "_")
• platform is linux_x86_64: sysconfig.get_platform().replace(".", "_").replace("-", "_")

Or the other way around:

>>> from packaging.tags import parse_tag
>>> t = list(parse_tag("pyston38-pyston_23_x86_64_linux_gnu-linux_x86_64"))[0]
>>> t.interpreter
'pyston38'
>>> t.abi
'pyston_23_x86_64_linux_gnu'
>>> t.platform
'linux_x86_64'

Hope this helps

[jelber2]

Ran into the same error as the OP. Any progress?

[dnicolodi]

I think #191 should solve this issue. Can someone running Pyston test?

[belm0]

@dnicolodi happy to try it, as I've just hit this issue today. But how do I go about having pip wheel scipy use a development branch of meson-python?

[dnicolodi]

IIRC pip wheel does not use an isolated environment for the build, thus you can install meson-python from the PR git branch in an virtual environment and execute pip wheel from there.

[rgommers]

I usually explicitly turn off build isolation:

# install the meson-python version you want to test, and other build dependencies
python -m build --no-isolation --skip-dependency-check

[belm0]

not sure what to make of this

I'm using pip wheel --no-build-isolation scipy==1.9.3 after installing meson-python from the branch.

      Installing /tmp/pip-wheel-06r0aanq/scipy_f0e7592b9c36444d945d8242174601c2/scipy/io/_harwell_boeing/tests/test_hb.py to /tmp/pip-wheel-06r0aanq/scipy_f0e7592b9c36444d945d8242174601c2/.mesonpy-5txqoc6k/install/usr/lib/python3.8-pyston2.3/site-packages/scipy/io/_harwell_boeing/tests
      Traceback (most recent call last):
        File "/home/venv/lib/python3.8-pyston2.3/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 144, in prepare_metadata_for_build_wheel
          hook = backend.prepare_metadata_for_build_wheel
      AttributeError: module 'mesonpy' has no attribute 'prepare_metadata_for_build_wheel'

      During handling of the above exception, another exception occurred:
...
        File "/home/venv/lib/python3.8-pyston2.3/site-packages/mesonpy/__init__.py", line 317, in _use_stable_abi
          abis.append(m.group('abi'))
      AttributeError: 'str' object has no attribute 'group'

[dnicolodi]

Obviously I introduced a bug with one of the last tweaks. Let me fix it. I'll be back to you ASAP

[belm0]

Thank you. Pyston is trivial to install from a .deb if you need it.

https://github.com/pyston/pyston/releases/tag/pyston_2.3.5