[BUG] scoped package bandwidth performance is poor and will just get worse in the future

[mikkorantalainen]

Current Behavior:

The existing API for scoped packages causes HUGE overhead for loading packages. The overhead is expected to raise in the future, if nothing is done.

Example: have dependency such as "@atlaskit/editor-core": "^112.44.8". To fulfill this, npm needs to fetch the editor-core package. This requires downloading 9789k of data just to figure out the URL for the package because this is a "scoped" package and as such, the URL http://registry.npmjs.org/@atlaskit/editor-core needs to be fully fetched just to acquire the URL for the actual package. If http://registry.npmjs.org/@atlaskit/editor-core/latest (or other more specific matches) were supported, there wouldn't be need to transfer so much data. Note that because the main URL must return full history for the package, the problem is expected to get worse in long run.

Expected Behavior:

$ curl http://registry.npmjs.org/@atlaskit/editor-core/latest
should work allowing clients to transfer lots of unneeded data.

Steps To Reproduce:

$ curl -I http://registry.npmjs.org/@atlaskit/editor-core/latest
HTTP/1.1 401 Unauthorized
...
npm-notice: ERROR: you cannot fetch versions for scoped packages
...
$ curl -I http://registry.npmjs.org/@atlaskit/editor-core/^112.44.8
HTTP/1.1 404 Not Found
...

curl http://registry.npmjs.org/@atlaskit/editor-core | wc -c
10024038

(or 9789k of data just to get the URL for the actual package which is 1911k)
curl https://registry.npmjs.org/@atlaskit/editor-core/-/editor-core-112.44.8.tgz | wc -c

Note that many packages have similar overhead: fetching the URL for specific version of the scoped package often takes 10x the data transfer of the full size for the actual package!

Environment:

Ubuntu 16.04.6 LTS
$ apt policy nodejs
...
Installed: 10.22.0-1nodesource1
...
500 https://deb.nodesource.com/node_10.x xenial/main amd64 Packages