RFC 3: using a dependency solver

[rth]

Here the idea is to use some standard Python dependency solver for creating the pyodide-lock.json from multiple sources (Pyodide distribution, PyPI, third party sources etc).

For instance it could look as follows,

1. Expose a Simple JSON API for the Pyodide distribution (so that Python package managers could understand it). This should already be done, but I can't find where (cc @ryanking13 )
2. Define a list of indexes with a priority . Same idea as --extra-index-url in pip
3. Run a standard Pythopn dependency solver on a list of input requirements, specifying emscripten*_wasm32 as the platform (or whatever it is now). Here the best tool remains to be determined,
   • using solvers from pip directly as pip-tools is doing.
   • https://github.com/sarugaku/resolvelib
   • others ?
4. Reformat the results in the pyodide-lock.json format (by taking information either from the original pyodide-lock.json)

Alternative proposals in #8 #9

[rth]

I'd prefer the second option, to create a separate lockfile from the original pyodide-lock file, [...], and I suspect that this would cause version conflicts between duplicate packages.

@ryanking13 Not necessarily, I think, if you have some priority between channels (--extra-index-url in pip pip option for instance) it should pick the version in the highest priority channel .

[ryanking13]

Expose a Simple JSON API for the Pyodide distribution (so that Python package managers could understand it). This should already be done, but I can't find where (cc @ryanking13 )

Right now this is done in pyodide-build/create_pypa_index.py, so it can be used in pip after calling pyodide venv. By the way, it uses Simple HTML API but I think we better change it to Simple JSON API.

[rth]

By the way, it uses Simple HTML API but I think we better change it to Simple JSON API.

Thanks! Absolutely. We cannot deploy HTML with the current CDN.

[ryanking13]

Among RFC 1, 2, and 3, IMO this is the best option for the long-term. I mean, in the future, we hope people can upload the Emscripten wheels to PyPI or to other package indices, and then people can create a lock file from it.

Then people who are okay with resolving dependencies dynamically can use Simple API (supported by micropip) to download packages from package indices, while people who make an application with fixed dependencies and don't want dependency resolution can create a lockfile.

[rth]

A Simple JSON API for this experimentation can be found in pyodide/pyodide#3979

[rth]

@juntyr This could also be a useful direction to explore.

The main part is, take an example list of requirements, some standard Python packaging tool (TBD), specify that the supported wheels are only of the form *cp311-emscripten_3_1_32_wasm32.whl or *py3-none-any.whl, point it to the Pyodide's Simple JSON API (above) + PyPI and ask it to generate a lock file (see the issue description above for more details). If you have any questions let me know.

For instance, jazzband/pip-tools#585 was essentially this feature request, but they decided it's out of scope for pip-compile.

Maybe @pradyunsg would also have some general advice or comments on this topic.

[joemarshall]

This is what pyodide build does - using resolvelib.

Incidentally, I do wonder how much this could be integrated into existing pyodide build functionality by making pyodide_build use this library to output a pyodide-lock.json. Or maybe just nick the logic from pypi.py in pyodide_build

e.g. thinking about use cases

1. You're building your own wheel from source (plus dependencies which are a combination of pypi and things built into pyodide) - pyodide build --build-dependencies does the correct resolution here, all it needs to do is generate a pyodide-lock.json instead of the current output which is a lockfile.

2. You want to include a wheel from pypi and all non-pyodide dependencies.
   pyodide build <pypi-version-spec> --build-dependencies does the resolution here, again needs to generate a lockfile.

3. You want to get multiple pypi wheels and dependencies
   pyodide build -r requirements.txt --build-dependencies

4. You have a bunch of wheels, and you want to create a lockfile that includes all of them and their dependencies
   pyodide build -r requirements.txt --build-dependencies
   with local paths to the wheels in the requirements.txt (right now I think local paths don't work, but is a trivial addition to the code.

[webknjaz]

Right now this is done in pyodide-build/create_pypa_index.py, so it can be used in pip after calling pyodide venv. By the way, it uses Simple HTML API but I think we better change it to Simple JSON API.

(with my PyPA hat on)
@ryanking13 could you rename PyPA (Python Packaging Authority) to PyPI (Python Packages Index) in that script? It's kinda confusing to see.

For instance, jazzband/pip-tools#585 was essentially this feature request, but they decided it's out of scope for pip-compile.

That's not entirely accurate, but is a rather complicated topic. That issue was closed automatically because of a linked documentation PR, while there are other issues centered around the same discussion, including mine. Here's a few comments I made there that might be of interest when we're talking about "lockfiles": jazzband/pip-tools#826 (comment) / jazzband/pip-tools#1326 (comment).

Though, is it just a single platform in case of pyodide? FWIW it might be a good idea to participate in reviving PEP 665.

[ryanking13]

@ryanking13 could you rename PyPA (Python Packaging Authority) to PyPI (Python Packages Index) in that script? It's kinda confusing to see.

Sure. Thanks for pointing that out.