Source distribution?

[Titus-von-Koeller]

I was just looking into some stuff around the PyPi file size limit and overall size limit for all files together. While looking at the releases, I realized that so far Tim was also publishing source distributions. What's strange is that source distributions, by their very nature, shouldn't contain pre-compiled binaries. But the size is almost the same and it does contain all binaries.

Not sure what to do about this. Do you guys have any opinion on this?

cc @matthewdouglas @rickardp @akx @wkpark

[matthewdouglas]

There's ways to put together a proper sdist, but until then, I do agree that it shouldn't have the prebuilt binaries included.

We would ideally want the sdist to invoke cmake and build the same way we do. There would be some work needed to make that happen.

[akx]

The sdist containing the precompiled binaries is a packaging bug, but it's not a big deal in the grand scheme of things. (That said, if all distributions are the same size, it sounds like all of them contain all binaries?)

As I said somewhere, we might not even want to push sdists at all to PyPI right now, since bnb is non-trivial to compile (even with the cmake stuff).

We'll want to first merge the pyproject.toml PR (sorry, on mobile, hard to look it up) and see that eg. hatch build -t wheel (if we choose Hatch for packaging, and I hope we do) ensures the binaries get bundled correctly.

[Titus-von-Koeller]

Btw, I submitted a request to increase our PyPi limit and am also looking into how much we're using of the total limit of 10GB for all artifacts (Tim has to look into that, he's the only one that can see that), because otherwise we might be unexpectedly not able to release at some point. I want to avoid that.

I suspect that we're not too far off the 10 GB: my thinking being that with source distro and built distro each having been around 100MB x 43 releases x 2 types of distros we might already be at ~8.5GB.. Just a rough, back of the envelope calculation.

[matthewdouglas]

It looks like they had previously bumped it to 200MB: pypi/support#3003

Is it possible to yank the sdists and some older releases?

[Titus-von-Koeller]

Wow, good catch! I wasn't aware of this and Tim didn't mention it either when I talked to him about this. Hmm, I wonder what to do about the request I did now. Should I just close it or is 300MB better to have more headroom?

[matthewdouglas]

Tim may not have noticed since it took over a month for a response :)
I hope there's no need for extra headroom, but maybe leave it open just in case?

[rickardp]

Source dists are probably not terribly useful at all at this point, until the .so files can be compiled from source using the Python builder (i.e. running cmake from the build backend). In either case, they should not contain any binaries. If they are it must be a bug. But I wasn't aware we even built source wheels?

[Titus-von-Koeller]

No, Tim was building source wheels in conjunction with the Makefile approach at the time. However, a couple of things were off in the packaging process, for example publishing BNB to be installable on any OS.

Yup, agree having had included the binaries in the sdist is definitely a bug. IMO, we don't need to publish sdists for the moment, but could be sth to look into for the future, jup, compiling .so files from source using the Python builder.

This is not urgent though and for now I wouldn't consider it high impact and hold off on it, unless someone thinks it's a fun little project and volunteers. I'd be happy to review and merge.

[matthewdouglas]

The recent v0.43.0 released without a sdist, and should be that way moving forward. For simpler packages I do think it's nice to have an sdist but I agree that it's not so useful when it's complicated to build and source + instructions are available elsewhere.

[Titus-von-Koeller]

Agreed that we don't need it for now. Might be a "nice to have" eventually if it's easy enough to implement.