Limit Request: mxnet* - 1G #243

szha · 2020-03-05T05:40:59Z

Project
mxnet, mxnet-native, mxnet-mkl, mxnet-cu100, mxnet-cu101, mxnet-cu102, mxnet-cu110, mxnet-cu111

Size of release
1GB

Which indexes
PyPI

Reasons for the request
Similar to #50, some of our variants are hitting size limit. Your help is most appreciated.

szha · 2020-03-05T05:41:58Z

We moved our nightly builds to https://dist.mxnet.io/python and we don't expect to release very often to these packages.

ewdurbin · 2020-03-05T14:33:24Z

@szha Thank you for moving your nightlies! That's a huge help for PyPI's bandwidth costs.

Would it be OK for PyPI Administrators to remove old nightly version from PyPI before approving and applying this limit increase?

ewdurbin · 2020-03-05T15:05:26Z

Current stats for the mxnet* Projects:

      name      | upload_limit | total_size | releases | release_files 
----------------+--------------+------------+----------+---------------
 mxnet          | 800 MB       | 36 GB      |      370 |          2247
 mxnet-cu100    | 600 MB       | 119 GB     |      173 |           277
 mxnet-cu100mkl | 600 MB       | 113 GB     |      162 |           246
 mxnet-cu101    | 800 MB       | 77 GB      |      128 |           175
 mxnet-cu101mkl | 800 MB       | 73 GB      |      123 |           155
 mxnet-cu102    | 800 MB       | 672 MB     |        1 |             1
 mxnet-cu80     | 400 MB       | 191 GB     |      209 |           325
 mxnet-cu80mkl  | 400 MB       | 178 GB     |      193 |           247
 mxnet-cu90     | 600 MB       | 175 GB     |      300 |           439
 mxnet-cu90mkl  | 600 MB       | 131 GB     |      217 |           320
 mxnet-cu92     | 800 MB       | 165 GB     |      264 |           382
 mxnet-cu92mkl  | 800 MB       | 120 GB     |      186 |           276
 mxnet-mkl      | 800 MB       | 77 GB      |      337 |          1884

ewdurbin · 2020-03-05T15:29:02Z

Cleaning up all existing nightlies on PyPI would be beneficial for us.

Here are the above stats split by nightly releases (matching version to `b\d\d\d\d\d\d\d\d$ ala 1.6.0b20200305)

Normal releases:

      name      | nightly | upload_limit | releases | release_files |  size   
----------------+---------+--------------+----------+---------------+---------
 mxnet          | f       | 800 MB       |       30 |           266 | 3506 MB
 mxnet-cu100    | f       | 600 MB       |        8 |            13 | 5186 MB
 mxnet-cu100mkl | f       | 600 MB       |        7 |            12 | 5277 MB
 mxnet-cu101    | f       | 800 MB       |        5 |             6 | 3054 MB
 mxnet-cu101mkl | f       | 800 MB       |        5 |             6 | 3225 MB
 mxnet-cu102    | f       | 800 MB       |        1 |             1 | 672 MB
 mxnet-cu80     | f       | 400 MB       |       29 |            58 | 11 GB
 mxnet-cu80mkl  | f       | 400 MB       |       25 |            35 | 9016 MB
 mxnet-cu90     | f       | 600 MB       |       22 |            35 | 12 GB
 mxnet-cu90mkl  | f       | 600 MB       |       20 |            26 | 9301 MB
 mxnet-cu92     | f       | 800 MB       |       13 |            22 | 8560 MB
 mxnet-cu92mkl  | f       | 800 MB       |       13 |            19 | 8092 MB
 mxnet-mkl      | f       | 800 MB       |       27 |           197 | 8267 MB

Nightlies:

 mxnet          | t       | 800 MB       |      340 |          1981 | 32 GB
 mxnet-cu100    | t       | 600 MB       |      165 |           264 | 114 GB
 mxnet-cu100mkl | t       | 600 MB       |      155 |           234 | 108 GB
 mxnet-cu101    | t       | 800 MB       |      123 |           169 | 74 GB
 mxnet-cu101mkl | t       | 800 MB       |      118 |           149 | 69 GB
 mxnet-cu80     | t       | 400 MB       |      180 |           267 | 72 GB
 mxnet-cu80mkl  | t       | 400 MB       |      168 |           212 | 63 GB
 mxnet-cu90     | t       | 600 MB       |      278 |           404 | 164 GB
 mxnet-cu90mkl  | t       | 600 MB       |      197 |           294 | 122 GB
 mxnet-cu92     | t       | 800 MB       |      251 |           360 | 156 GB
 mxnet-cu92mkl  | t       | 800 MB       |      173 |           257 | 112 GB
 mxnet-mkl      | t       | 800 MB       |      310 |          1687 | 69 GB

So removing old nightlies for all these projects would reduce PyPI mirror sizes by about 1.1TB.

szha · 2020-03-10T05:58:01Z

@ewdurbin yes we are happy to move them. It would be great if we could archive them in our S3 repo. Is there a way to get their s3 keys/links? (s3 key is preferred)

ewdurbin · 2020-03-10T13:30:04Z

@szha Here is a CSV containing all existing nightlies including the name, version, filename, and URL of the file: https://gist.github.com/ewdurbin/72804a663e073f3967474d9ccd753c23

ewdurbin · 2020-03-10T17:49:32Z

It looks like mxnet and mxnet-mkl are still publishing nightlies to PyPI. Will these be moved to https://dist.mxnet.io/python as well?

szha · 2020-03-15T17:18:57Z

@ewdurbin Thanks for the list. Yes. It's publishing from an old pipeline and we will move their nightly builds to S3 only too.

In the meantime, I'm archiving all the nightly builds to our s3 and it may take a while. I will report back once it finishes and when the wheels are safe to delete.

ewdurbin · 2020-03-15T22:31:10Z

Thank you @szha!

szha · 2020-03-16T15:20:24Z

All done on the migration. Let me check with the projects in the ecosystem and ask them to migrate for their CI.

ewdurbin · 2020-03-18T14:11:11Z

Thank you @szha, is it OK for us to remove the existing nightly releases then?

ewdurbin · 2020-03-31T10:45:51Z

@szha checking in on the status of removing the old nightlies before granting this limit request.

szha · 2020-03-31T17:03:30Z

@ewdurbin let's wait until the end of the week for the projects to update to avoid disruption. Thanks.

ewdurbin · 2020-04-09T11:31:18Z

@szha another check-in awaiting confirmation for removal of old nightlies.

ewdurbin · 2020-04-30T12:26:42Z

@szha still awaiting confirmation to remove the old nightlies from PyPI before bumping the limits requested.

szha · 2020-04-30T16:31:11Z

@ewdurbin thanks for the reminder. I will do another round of check-in with these projects on the status. Also, we will have some more requests coming in due to the upcoming new release of CUDA. I will ping here once I finish the above

szha · 2020-05-02T01:35:17Z

Updated requests with larger size limit request and fewer packages. Also the nightly builds are safe to remove now. Thanks.

szha · 2020-05-30T03:34:54Z

Any update?

szha · 2020-07-08T19:56:04Z

Ping @ewdurbin

ewdurbin · 2020-07-08T20:00:25Z

Bleh, thank you for the ping @szha! The last month has been wild.

In the meantime, some new updates to how limits are handled were introduced. I need to do the following:

Remove the agreed releases
Calculate current usage based on new project wide limits and share here to decide on an appropriate limit
Set new project wide limits for the projects in question based on the new agreed upon limit
Close this issue

szha · 2020-07-26T22:19:57Z

@ewdurbin gentle ping on this.

szha · 2020-08-28T07:06:17Z

Pinging on this again as it has been another month... @ewdurbin @di

ewdurbin · 2020-08-28T10:03:08Z

All old nightlies have been removed, here is full log of what was performed:

https://gist.github.com/ewdurbin/61428c2259057f5e466fa8289ae829f5

ewdurbin · 2020-08-28T10:22:48Z

Individual file limits have been raised to 1024MB, which is the absolute maximum that we can accommodate at this time.

Current status for all projects owned by mxnet or mxnet-ci users on PyPI with releases:

        name         | releases | release_files | project_total_size | project_total_size_limit | upload_limit | upload_limit 
---------------------+----------+---------------+--------------------+--------------------------+--------------+--------------
 gluonnlp            |       25 |            25 | 4475 kB            |                          |              |             
 mxnet               |       31 |           267 | 3558 MB            |                          | 1024 MB      |   1073741824
 mxnet-cu100         |        8 |            13 | 5186 MB            |                          | 1024 MB      |   1073741824
 mxnet-cu100mkl      |        7 |            12 | 5277 MB            |                          | 1024 MB      |   1073741824
 mxnet-cu101         |        5 |             6 | 3055 MB            |                          | 1024 MB      |   1073741824
 mxnet-cu101mkl      |        5 |             6 | 3226 MB            |                          | 1024 MB      |   1073741824
 mxnet-cu102         |        1 |             1 | 673 MB             |                          | 1024 MB      |   1073741824
 mxnet-cu102mkl      |        1 |             1 | 716 MB             |                          | 1024 MB      |   1073741824
 mxnet-cu75          |       20 |            23 | 2733 MB            |                          | 1024 MB      |   1073741824
 mxnet-cu75mkl       |       15 |            18 | 2769 MB            |                          | 1024 MB      |   1073741824
 mxnet-cu80          |       29 |            58 | 11 GB              | 20 GB                    | 1024 MB      |   1073741824
 mxnet-cu80mkl       |       25 |            35 | 9016 MB            | 20 GB                    | 1024 MB      |   1073741824
 mxnet-cu90          |       22 |            35 | 12 GB              | 20 GB                    | 1024 MB      |   1073741824
 mxnet-cu90mkl       |       20 |            26 | 9301 MB            | 20 GB                    | 1024 MB      |   1073741824
 mxnet-cu91          |        7 |            12 | 4183 MB            |                          | 1024 MB      |   1073741824
 mxnet-cu91mkl       |        5 |             6 | 2118 MB            |                          | 1024 MB      |   1073741824
 mxnet-cu92          |       14 |            23 | 9315 MB            | 20 GB                    | 1024 MB      |   1073741824
 mxnet-cu92mkl       |       13 |            19 | 8093 MB            | 20 GB                    | 1024 MB      |   1073741824
 mxnet-mkl           |       27 |           197 | 8267 MB            | 20 GB                    | 1024 MB      |   1073741824
 mxnet-tensorrt-cu90 |        1 |             3 | 184 MB             |                          | 1024 MB      |   1073741824
 mxnet-tensorrt-cu92 |        1 |             2 | 133 MB             |                          | 1024 MB      |   1073741824
(21 rows)

The default Project total size limit is 10GB.

What cadence is anticipated for releases? That can help us set an initial limit for project total size across these projects.

szha · 2020-08-28T15:03:45Z

Thanks! We are at a pace of about a release every Three months though we want to bring it down to 1 month. The release will only happen to a few of them and we likely won’t be making new releases to the cu75/cu80/cu91 projects or the *mkl projects.

GluonNLP is Python-only so it won’t need adjustments.

szha · 2020-08-28T15:05:56Z

By the way did you raise the limit of mxnet-cu110/111? We registered those packages but they aren’t showing up in the table above (likely due to lack of releases)

ewdurbin · 2020-08-28T15:33:40Z

yes, sorry limits for mxnet-cu110 and mxnet-cu111 have been raised to 1024 MB as well and were not included for the reason you specified (no releases).

ewdurbin · 2020-08-28T15:34:46Z

I'm going to bump the total project size limit for all of the MXnet projects with 1024MB upload limits to 24GB. We can discuss again when we approach that limit.

ewdurbin · 2020-08-28T15:41:05Z

Here is the current status for all the projects:

        name         | _total_size_limit | _upload_limit 
---------------------+-------------------+---------------
 gluonasr            |                   | 
 gluonnlp            |                   | 
 mxnet               | 24 GB             | 1024 MB
 mxnet-cpu           |                   | 
 mxnet-cu100         | 24 GB             | 1024 MB
 mxnet-cu100mkl      | 24 GB             | 1024 MB
 mxnet-cu101         | 24 GB             | 1024 MB
 mxnet-cu101mkl      | 24 GB             | 1024 MB
 mxnet-cu102         | 24 GB             | 1024 MB
 mxnet-cu102mkl      | 24 GB             | 1024 MB
 mxnet-cu110         | 24 GB             | 1024 MB
 mxnet-cu111         | 24 GB             | 1024 MB
 mxnet-cu75          | 24 GB             | 1024 MB
 mxnet-cu75mkl       | 24 GB             | 1024 MB
 mxnet-cu80          | 24 GB             | 1024 MB
 mxnet-cu80mkl       | 24 GB             | 1024 MB
 mxnet-cu90          | 24 GB             | 1024 MB
 mxnet-cu90mkl       | 24 GB             | 1024 MB
 mxnet-cu91          | 24 GB             | 1024 MB
 mxnet-cu91mkl       | 24 GB             | 1024 MB
 mxnet-cu92          | 24 GB             | 1024 MB
 mxnet-cu92mkl       | 24 GB             | 1024 MB
 mxnet-gpu           |                   | 
 mxnet-mkl           | 24 GB             | 1024 MB
 mxnet-native        |                   | 
 mxnet-tensorrt-cu90 | 24 GB             | 1024 MB
 mxnet-tensorrt-cu92 | 24 GB             | 1024 MB

szha · 2020-08-28T16:38:45Z

awesome. thank you for the help @ewdurbin !

pradyunsg · 2020-08-31T14:42:16Z

@ewdurbin I took the liberty of updating #243 (comment). :)

szha added the limit request label Mar 5, 2020

di assigned ewdurbin Apr 30, 2020

di added the status: awaiting response Needs more information before proceeding label Apr 30, 2020

szha changed the title ~~Limit Request: mxnet* - 850MB~~ Limit Request: mxnet* - 1G May 2, 2020

ewdurbin closed this as completed Aug 31, 2020

szha mentioned this issue Nov 23, 2020

Limit Request: mx-cu* - 1G #755

Closed

szha mentioned this issue Mar 21, 2021

Limit Request: mxnet-cu112 - 1G #970

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Limit Request: mxnet* - 1G #243

Limit Request: mxnet* - 1G #243

szha commented Mar 5, 2020 •

edited

Loading

szha commented Mar 5, 2020

ewdurbin commented Mar 5, 2020

ewdurbin commented Mar 5, 2020 •

edited

Loading

ewdurbin commented Mar 5, 2020

szha commented Mar 10, 2020

ewdurbin commented Mar 10, 2020

ewdurbin commented Mar 10, 2020

szha commented Mar 15, 2020 •

edited

Loading

ewdurbin commented Mar 15, 2020

szha commented Mar 16, 2020

ewdurbin commented Mar 18, 2020

ewdurbin commented Mar 31, 2020

szha commented Mar 31, 2020

ewdurbin commented Apr 9, 2020

ewdurbin commented Apr 30, 2020

szha commented Apr 30, 2020

szha commented May 2, 2020

szha commented May 30, 2020

szha commented Jul 8, 2020

ewdurbin commented Jul 8, 2020 •

edited by pradyunsg

Loading

szha commented Jul 26, 2020

szha commented Aug 28, 2020

ewdurbin commented Aug 28, 2020

ewdurbin commented Aug 28, 2020 •

edited

Loading

szha commented Aug 28, 2020

szha commented Aug 28, 2020

ewdurbin commented Aug 28, 2020

ewdurbin commented Aug 28, 2020

ewdurbin commented Aug 28, 2020

szha commented Aug 28, 2020

pradyunsg commented Aug 31, 2020

Limit Request: mxnet* - 1G #243

Limit Request: mxnet* - 1G #243

Comments

szha commented Mar 5, 2020 • edited Loading

szha commented Mar 5, 2020

ewdurbin commented Mar 5, 2020

ewdurbin commented Mar 5, 2020 • edited Loading

ewdurbin commented Mar 5, 2020

szha commented Mar 10, 2020

ewdurbin commented Mar 10, 2020

ewdurbin commented Mar 10, 2020

szha commented Mar 15, 2020 • edited Loading

ewdurbin commented Mar 15, 2020

szha commented Mar 16, 2020

ewdurbin commented Mar 18, 2020

ewdurbin commented Mar 31, 2020

szha commented Mar 31, 2020

ewdurbin commented Apr 9, 2020

ewdurbin commented Apr 30, 2020

szha commented Apr 30, 2020

szha commented May 2, 2020

szha commented May 30, 2020

szha commented Jul 8, 2020

ewdurbin commented Jul 8, 2020 • edited by pradyunsg Loading

szha commented Jul 26, 2020

szha commented Aug 28, 2020

ewdurbin commented Aug 28, 2020

ewdurbin commented Aug 28, 2020 • edited Loading

szha commented Aug 28, 2020

szha commented Aug 28, 2020

ewdurbin commented Aug 28, 2020

ewdurbin commented Aug 28, 2020

ewdurbin commented Aug 28, 2020

szha commented Aug 28, 2020

pradyunsg commented Aug 31, 2020

szha commented Mar 5, 2020 •

edited

Loading

ewdurbin commented Mar 5, 2020 •

edited

Loading

szha commented Mar 15, 2020 •

edited

Loading

ewdurbin commented Jul 8, 2020 •

edited by pradyunsg

Loading

ewdurbin commented Aug 28, 2020 •

edited

Loading