Limit Request: mxnet-* - 800MB

[szha]

The name of the project: mxnet, mxnet-mkl, mxnet-cu92, mxnet-cu92mkl, mxnet-cu101, mxnet-cu101mkl, mxnet-cu102, mxnet-cu102mkl
The size of your release, in megabytes: 800mb
Which index/indexes you need the increase for (PyPI, Test PyPI, or both): both

Please help me increase the upload limit, thank you!

[eric-haibin-lin]

any update on this?

[szha]

Hi @jamadden. Your continued support and help in this matter would be most appreciated. Thanks!

[apeforest]

Would it be possible to have an ETA for this request?

[di]

Hi folks, this is a really big ask. The mxnet-* projects already represent a huge portion of PyPI's total size on disk and in terms of bandwidth. Per https://pypi.org/stats/, the mxnet-* projects total more than 1.5TB of PyPI's 6.5TB total size.

Is there anything that can be done to reduce either the size of these releases, or the frequency of them?

[szha]

We have done a couple of things so far:

• we cleaned up the nightly release artifacts and kept one artifact per week for artifacts with longer age.
• we discontinued the versions for older cuda variants such as cuda 7.5, cuda 8.0.
• we started exploring binary compression with upx.

We plan to:

• clean up all nightly builds for discontinued packages and keep only the release versions.
• discuss in the community to reduce the number of variants supported.
• make the *mkl variants the default and reduce the supported variants by half.
• architectural changes to reduce the binary size.

Given that these efforts take time, we appreciate it if you could raise the limits to unblock us for now. Our ask is not with respect to all mxnet-*, but only this subset: mxnet, mxnet-mkl, mxnet-cu92, mxnet-cu92mkl, mxnet-cu101, mxnet-cu101mkl, mxnet-cu102, mxnet-cu102mkl.

[szha]

@di would that address your concern? Could you unblock us?

[di]

Cleaning up the releases after they're made doesn't really change much here: the biggest cost for PyPI from these distributions is the bandwidth consumed when several hundred mirrors attempt to mirror each release immediately after it's published.

We need to either reduce either the size of these releases, or the frequency of them.

For example, most of these projects are releasing several hundred MB prereleases almost nightly. Is this necessary? Could they switch to weekly releases?

[szha]

@di I will suggest that to the community. Realistically, we may want to keep at minimum one variant that releases nightly. This is to make sure that the ecosystem projects can expose problems at a faster pace. I hope that's understandable.

[szha]

On a side note, Google's Tensorflow has the tf-nightly* packages which total more than 1.65TB. Do you happen to know what the plan is there @di ?

[di]

That project currently has a upload limit of 500MB. We don't currently plan to increase it (or any of the other top-100 projects) without asking them to do the same.

[leezu]

I started a discussion on the MXNet developer list to stop releasing nightly version to Pypi and just publish them on a MXNet project-owned S3 bucket: https://lists.apache.org/thread.html/cba5e34a59af5e3ce99bbad912341efef32fb409c022d9239aa2994a@%3Cdev.mxnet.apache.org%3E

@di If there are no objections from the MXNet community, we would only release releases and release candidates to Pypi, which should significantly reduce your cost.
Would you be willing to increase the upload limit in that case?

[di]

Yes, I think that would be acceptable. Thanks for being accommodating.

[leezu]

We reached consensus on the mailinglist to stop uploading nightly releases to PyPI. https://lists.apache.org/thread.html/d424380b145d7c2329e4bb3039595cc6ae4dcb99381522fd6c1a0383%40%3Cdev.mxnet.apache.org%3E

@di when do you expect to be able to process the limit increase?

[pradyunsg]

I just increased the limits as detailed below. Please be mindful of how frequently you make releases with such a high limit -- each release will have a not-insignificant impact on how much traffic PyPI has to serve.

---

Raised to 800 MB, on both PyPI and TestPyPI:

• mxnet

Raised to 800 MB on PyPI, does not exist on TestPyPI:

• mxnet-mkl
• mxnet-cu92
• mxnet-cu92mkl
• mxnet-cu101
• mxnet-cu101mkl
• mxnet-cu102
• mxnet-cu102mkl

[di]

Thanks @pradyunsg