Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Please provide PEP 503 compliant indices for CUDA versions of packages #5410

Open
Froskekongen opened this issue Jan 14, 2021 · 10 comments
Open

Please provide PEP 503 compliant indices for CUDA versions of packages #5410

Froskekongen opened this issue Jan 14, 2021 · 10 comments
Assignees
@skye @yashk2810
Labels
contributions welcome The JAX team has not prioritized work on this. Community contributions are welcome. enhancement New feature or request NVIDIA GPU Issues specific to NVIDIA GPUs open Issues intentionally left open, with no schedule for next steps. P1 (soon) Assignee is working on this now, among other tasks. (Assignee required)

Comments

@Froskekongen
Copy link

At https://storage.googleapis.com/jax-releases/jax_releases.html, we can find cuda versions of jaxlib. This index is not PEP 503-compliant, and can therefore not be used in a package manager such as poetry.

Can you please provide an index that respects the requirements of PEP 503?
@skye
Copy link
Collaborator

skye commented Jan 14, 2021

Are you able to say more about what's not compliant about it, or what kind of error you're getting? I can believe we messed something up here, but it's a little hard to pinpoint exactly what, especially since the index works with pip.

@skye skye self-assigned this Jan 14, 2021
@hawkinsp
Copy link
Collaborator

We didn't necessarily intend that our repository be PEP 503 compliant, just that pip understands it.

To be PEP 503 compliant this I think we'd need to serve our wheels a different way. e.g. "Within a repository, the root URL (/ for this PEP which represents the base URL) MUST be a valid HTML5 page with a single anchor element per project in the repository." is something that cannot be satisfied by a raw GCS bucket. This is certainly possible, just a bit of work to move our web hosting around.

@hawkinsp hawkinsp added the enhancement New feature or request label Jan 15, 2021
@Froskekongen
Copy link
Author

To be able to use gpu versions with package managers in a reasonable way, I think using a format that is compatible with pypi would be greatly appreciated. Here is an example where a third party provided something that is compatible for pytorch: https://vxlabs.com/pypi/

Having a simple page with this structure that refers to the already existing wheels should be enough, no?

I guess being pypi compatible is not necessarily the same as being PEP 503 compliant.

@skye
Copy link
Collaborator

skye commented Jan 15, 2021

@hawkinsp do you have any ideas as to where we could (easily) host such a page?

I'm gonna mark this contributions welcome for now, since it sounds like anyone can set up such a page and link to the wheels in our GCS bucket.

@skye skye added contributions welcome The JAX team has not prioritized work on this. Community contributions are welcome. open Issues intentionally left open, with no schedule for next steps. labels Jan 15, 2021
@skye skye removed their assignment Jan 15, 2021
@JanLuca
Copy link

JanLuca commented Sep 20, 2021

You could just host a GitHub pages with the corresponding PEP 503 structure. I just wrote a script to generate a index.html for this purpose:

#!/usr/bin/env python3

import re
import requests
import textwrap

TEMPLATE = """<!DOCTYPE html>
<html>
  <head>
    <meta name="pypi:repository-version" content="1.0">
    <title>Links for jaxlib</title>
  </head>
  <body>
    <h1>Links for jaxlib</h1>
{links}
    </body>
</html>
"""

TEMPLATE_HREF = '<a href="{url}">{name}</a>'

re_a_href = re.compile('<a[^>]*?href="(?P<url>[^"]*?)"[^>]*?>(?P<name>[^<]*?)</a>')


def main():
    r = requests.get("https://storage.googleapis.com/jax-releases/jax_releases.html")

    result = []

    for m in re_a_href.finditer(r.text):
        name = m.group("name").split("/")[-1]

        result.append(TEMPLATE_HREF.format(name=name, url=m.group("url")))

    links = textwrap.indent("\n".join(result), "    ")

    index = TEMPLATE.format(links=links)

    with open("index.html", "w") as f:
        f.write(index)


if __name__ == "__main__":
    main()

This can be used to update a simple pip index interface, I just tested it here: https://janluca.github.io/pypi/
The source code can be found in https://github.com/JanLuca/janluca.github.io

Would it be possible to just create some jax GitHub page in the organization for this purpose?

@JanLuca
Copy link

JanLuca commented Oct 4, 2021

@skye Would it be possible to have such a GitHub page hosted along the jax repository?

@skye
Copy link
Collaborator

skye commented Oct 4, 2021

I thought GitHub pages were only per-user/organization, but it looks like you can have per-project pages as well. So yes, this does seem like it should be possible. I can look into this, thanks for providing the example script!

@jorenham
Copy link

jorenham commented Jan 12, 2022 •
edited
Loading

I have created a daily-rebuilding repository for this, see https://github.com/jorenham/jax_pep503. With Poetry, you can use it like:

[tool.poetry.dependencies]
...
jax = {extras = ["cuda"], version = "^0.3.13"}

[[tool.poetry.source]]
name = "jorenham/jax_pep503"
url = "https://jorenham.github.io/jax_pep503/"
secondary = true

@CactusPuppy
Copy link

With pypa/pip#10825, the lack of a <!DOCTYPE html> tag will potentially start to prevent pip from understanding the index page. Please consider adding a DOCTYPE tag before the html tag to ensure operability with future versions of pip.

@jakevdp jakevdp mentioned this issue Mar 4, 2022
Closed
@abn
Copy link

abn commented May 8, 2022

Just a drive by comment on this coming from python-poetry/poetry#5516. We have implmented single page support in python-poetry/poetry#5517. However, for tools like Poetry, since metadata is important when resolving a project's universal dependencies, having package checksums (described in PEP 503) and PEP 658 support enabled would considerably help the dependency resolution and lockfile generation.

@ColdTeapot273K ColdTeapot273K mentioned this issue May 28, 2022
Closed
2 tasks
@warmlogic warmlogic mentioned this issue Jun 24, 2022
Merged
@sudhakarsingh27 sudhakarsingh27 added NVIDIA GPU Issues specific to NVIDIA GPUs P1 (soon) Assignee is working on this now, among other tasks. (Assignee required) labels Aug 10, 2022
@jakevdp jakevdp mentioned this issue Apr 7, 2023
Closed
@daquinteroflex daquinteroflex mentioned this issue Dec 14, 2023
Closed
@NeilGirdhar NeilGirdhar mentioned this issue Mar 17, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@skye skye

@yashk2810 yashk2810

Labels
contributions welcome The JAX team has not prioritized work on this. Community contributions are welcome. enhancement New feature or request NVIDIA GPU Issues specific to NVIDIA GPUs open Issues intentionally left open, with no schedule for next steps. P1 (soon) Assignee is working on this now, among other tasks. (Assignee required)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
9 participants
@skye @abn @hawkinsp @JanLuca @Froskekongen @sudhakarsingh27 @jorenham @CactusPuppy @yashk2810