[npm node] support for custom npm registry #1277

iamstarkov · 2017-11-18T01:23:18Z

there are two types of custom npm registries:

public: due to geographical or maybe political reasons https://registry.npm.taobao.org/—essentially copy of https://registry.npmjs.org/
private: due to legal requirements (my case) or redundancy reasoning—either npm enterprise or sinopia

affected endpoints:

npm node version integration now supports /npm/https/registry.npm.taobao.org/v/express.svg:

-/^\/npm\/v\/(?:@([^/]+))?\/?([^/]*)\/?([^/]*)\.(svg|png|gif|jpg|json)$/
+/^\/npm(?:\/(http(?:s)?)\/([^/]+))?\/v\/(?:@([^/]+))?\/?([^/]*)\/?([^/]*)\.(svg|png|gif|jpg|json)$/

npm license integration now supports /npm/https/registry.npm.taobao.org/l/express.svg:

-/^\/npm\/l\/(?:@([^/]+)\/)?([^/]+)\.(svg|png|gif|jpg|json)$/
+/^\/npm(?:\/(http(?:s)?)\/([^/]+))?\/l\/(?:@([^/]+)\/)?([^/]+)\.(svg|png|gif|jpg|json)$/

npm node version integration now supports /node/https/registry.npm.taobao.org/v/express.svg:

-/^\/npm\/l\/(?:@([^/]+)\/)?([^/]+)\.(svg|png|gif|jpg|json)$/
+/^\/npm(?:\/(http(?:s)?)\/([^/]+))?\/l\/(?:@([^/]+)\/)?([^/]+)\.(svg|png|gif|jpg|json)$/

downloads related endpoints are not affected due to custom https://api.npmjs.org/ server, which complicates stuff. But apart from complexity, there are several reasons not to modify these endpoints:

for public registries its (maybe) okay to use default registry
for private registries this downloads metric is (maybe) unnecessary

P.S. inspiration for /npm(/:protocol/:host)/… is jenkins endpoint

extend routes to optionally match protocol (http or https) and host. utilise ./lib/npm-provider::getNpmRegistryUrl to construct custom npm registry urls or fallback to standard one.

paulmelnikow · 2017-11-18T17:11:42Z

Thanks for this!

I appreciate that you're modeling it the Jenkins badge. Since #1186 it's possible to pass arbitrary parameters through the query string. I'd like to use this for exception / rarely used options, long options, or options which require munging to fit into the URL. This seems like a good place for it. Basically, something like ?registry_uri=.

The benefit is a more consistent and understandable URL pattern. The resultant URL isn't quite as pretty and takes a bit of effort to encode, but it's also not going to be used by many people, and after #701 it should be really easy to encode one of these through the UI.

You can see a few examples in server.js where an array of query string field names is passed in. This is from the dynamic JSON badge:

queryParams: ['uri', 'query', 'prefix', 'suffix'],

paulmelnikow · 2017-11-18T17:13:04Z

lib/all-badge-examples.js

+      },
+      {
+        title: 'custom registry :: npm (scoped with tag)',
+        previewUri: '/npm/https/registry.npm.taobao.org/v/@cycle/core/canary.svg',


How about choosing one of these four and removing the others? The title should be something like npm (scoped with tag, custom registry). Same with the node badges. I feel like people will understand the pattern.

it looks like this now, hope its okay

paulmelnikow · 2017-11-18T17:14:32Z

lib/npm-provider.js

@@ -36,6 +36,15 @@ function mapNpmDownloads(camp, urlComponent, apiUriComponent) {
  }));
 }

+// npm registry url
+function getNpmRegistryUrl(maybeProtocol, maybeHost) {


Using the query string should make this a little simpler. If you still need this function it should go in a new file npm-badge-helpers.js.

what is the difference between npm-provider and npm-badge-helpers?

Most of the badge providers are in server.js, though a few, like npm, are pulled out. We use -provider to refer to the function which implements the badge.

The -badge-helpers are service-specific utility functions which are invoked by the providers.

Yea, in a couple cases the line is gray. Though this one seems to be a utility function.

and i got rid off of helper function, now i only use default npm registry uri as a constant to share

paulmelnikow · 2017-11-18T17:25:01Z

server.js

  } else {
    // e.g. https://registry.npmjs.org/@cedx%2Fgulp-david
    // because https://registry.npmjs.org/@cedx%2Fgulp-david/latest does not work
    const path = encodeURIComponent(`${scope}/${packageName}`);
-    apiUrl = `https://registry.npmjs.org/@${path}`;
+    apiUrl = `${registryUrl}/@${path}`;


I think I wrote tests for the current Node and NPM badges. Could you add tests of the new routes?

iamstarkov · 2017-11-18T17:43:21Z

no problems. i'll change stuff. just to be on the same page. here this is the list to change/add:

remove extra examples
move helper to npm-badge-helpers.js
query params instead of complicating the url pattern
tests

if im not right, pls correct me

iamstarkov · 2017-11-18T18:10:33Z

and thanks for the feedback

paulmelnikow · 2017-11-18T21:24:47Z

Yup, that's it. And if you keep the helper function, please move it to a different file.

Thanks!

iamstarkov · 2017-11-19T02:59:43Z

only tests left

iamstarkov · 2017-11-19T04:04:12Z

done

iamstarkov · 2017-11-19T04:12:18Z

because of added queryParams and following added indentation to handler diff looks messy.

to make things simpler to review use diff url without whitespace diff https://github.com/badges/shields/pull/1277/files?w=1

iamstarkov · 2017-11-19T04:29:17Z

tests are ok, its just ping to chinese registry is very long/big and mocha is timing out.

what do you think?

paulmelnikow · 2017-11-19T16:26:17Z

Excellent! Thanks for the diff; it's easy enough to read. This looks great.

Only one thing I'd ask for, is tests of the license badge. It seems think we have any now. The reason it seems important is that all that code was in flight. You could just pick a package and hard-code the expected value.

Is there another mirror that's faster to reach from the U.S.? Alternatively we could just leave the flaky test. I've solved the timeout issue in IcedFrisby; it's waiting to be shipped in 2.0.

iamstarkov · 2017-11-19T22:06:41Z

sure, i'll add tests for license service

iamstarkov · 2017-11-19T22:12:20Z

about registries

iamstarkov · 2017-11-19T22:16:42Z

its complicated. there used to be some semi-supported mirrors. Nowadays its only registry.npm.taobao.org and r.cnpmjs.org with average ping 275ms and 425ms respectively. So if taobao isnt a good fit, then r.cnpmjs isnt either.

iamstarkov · 2017-11-19T22:19:09Z

~/projects/oss/shields feat/custom-npm-registry
☯ npm i -g nrm -dddddd
npm info it worked if it ends with ok
npm info using npm@5.5.1
npm info using node@v8.9.1
npm http fetch GET 304 https://registry.npmjs.org/nrm 938ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/async 96ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/ini 125ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/node-echo 125ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/extend 129ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/request 146ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/npm 228ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/only 254ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/open 784ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/commander 797ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/mkdirp 60ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/coffee-script 139ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/jistype 990ms (from cache)

from this log default registry is registry.npmjs.org. but there is as well registry.npmjs.org which is basically a dns alias to default one:

☯ ping registry.npmjs.org -c 11
PING a.sni.fastly.net (151.101.84.162): 56 data bytes
avg 48ms

☯ ping registry.npmjs.com -c 11
PING a.sni.fastly.net (151.101.84.162): 56 data bytes
avg 11 ms

iamstarkov · 2017-11-19T22:19:44Z

i'll use registry.npmjs.com, because its not default one, but fast

….npmjs.com in examples and tests

iamstarkov · 2017-11-19T22:38:34Z

yay, registry.npmjs.com fixed it

iamstarkov · 2017-11-19T23:10:51Z

done

iamstarkov · 2017-11-19T23:11:29Z

if nothing left to fix, this pull-request should be fine. @paulmelnikow thanks for your input, feedback and patience

paulmelnikow · 2017-11-21T00:22:57Z

Thanks for all your work!

iamstarkov · 2017-11-21T10:10:54Z

hurray

iamstarkov · 2017-11-21T10:10:57Z

thanks

iamstarkov · 2017-11-21T15:45:55Z

will you release it to https://shields.io/ anytime soon?

paulmelnikow · 2017-11-21T18:38:08Z

I can't; I don't have access. Deploys usually happen every 1–3 weeks. Thaddée, who has limited time on this project, is the only sysadmin. He's working on giving me access to deploy and logs, but doing so is complicated because the hosting account (and maybe the servers too) are shared with other services he runs.

iamstarkov added 3 commits November 18, 2017 02:22

[npm] add badges examples with custom npm registry

762e483

[npm] add helper to get npm registry url

9eddf38

[npm] implement custom npm registry support

5a54d3a

extend routes to optionally match protocol (http or https) and host. utilise ./lib/npm-provider::getNpmRegistryUrl to construct custom npm registry urls or fallback to standard one.

paulmelnikow reviewed Nov 18, 2017

View reviewed changes

paulmelnikow added the service-badge New or updated service badge label Nov 18, 2017

[npm] reduce amount of examples

733c65b

iamstarkov force-pushed the feat/custom-npm-registry branch from b5cc1f1 to 733c65b Compare November 19, 2017 01:09

iamstarkov added 3 commits November 19, 2017 02:36

[npm] move getNpmRegistryUrl to ./lib/npm-badge-helpers

21ed1a2

[npm] keep only defaultNpmRegistryUri as a constant reference

90352c5

[npm] rework custom registry support to utilise query params

c16c007

[npm] add service-tests for custom registry support

6c71b35

iamstarkov force-pushed the feat/custom-npm-registry branch from efe8a76 to 6c71b35 Compare November 19, 2017 04:10

paulmelnikow changed the title ~~[npm] support for custom npm registry~~ [npm node] support for custom npm registry Nov 19, 2017

[npm] change custom registry from registry.npm.taobao.org to registry…

852dc6a

….npmjs.com in examples and tests

[npm] add tests for a license part

90f8899

paulmelnikow merged commit e588bef into badges:master Nov 21, 2017

iamstarkov deleted the feat/custom-npm-registry branch November 21, 2017 10:10

dependabot-preview bot mentioned this pull request Apr 15, 2019

Bump eslint-plugin-import from 2.16.0 to 2.17.1 #3313

Merged

paulmelnikow mentioned this pull request Jul 16, 2019

Finish moving hostnames from routes to query strings #3714

Closed

7 tasks

This was referenced Apr 26, 2021

[Snyk] Security upgrade redis from 2.6.5 to 3.1.1 dineshvgp/shields#15

Open

[Snyk] Security upgrade redis from 2.6.5 to 3.1.1 mkg20001/shields#11

Open

[Snyk] Security upgrade redis from 2.8.0 to 3.1.1 Richienb/shields#15

Closed

Richienb mentioned this pull request Oct 5, 2022

[Snyk] Fix for 8 vulnerabilities Richienb/shields#23

Closed

snyk-bot mentioned this pull request Oct 6, 2022

[Snyk] Fix for 8 vulnerabilities Richienb/shields#24

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[npm node] support for custom npm registry #1277

[npm node] support for custom npm registry #1277

iamstarkov commented Nov 18, 2017 •

edited

Loading

paulmelnikow commented Nov 18, 2017

paulmelnikow Nov 18, 2017

iamstarkov Nov 19, 2017

iamstarkov Nov 19, 2017

paulmelnikow Nov 18, 2017

iamstarkov Nov 18, 2017

paulmelnikow Nov 18, 2017

iamstarkov Nov 19, 2017

iamstarkov Nov 19, 2017

paulmelnikow Nov 18, 2017

iamstarkov Nov 18, 2017

iamstarkov Nov 19, 2017

iamstarkov commented Nov 18, 2017 •

edited

Loading

iamstarkov commented Nov 18, 2017

paulmelnikow commented Nov 18, 2017

iamstarkov commented Nov 19, 2017

iamstarkov commented Nov 19, 2017

iamstarkov commented Nov 19, 2017

iamstarkov commented Nov 19, 2017 •

edited

Loading

paulmelnikow commented Nov 19, 2017

iamstarkov commented Nov 19, 2017

iamstarkov commented Nov 19, 2017

iamstarkov commented Nov 19, 2017

iamstarkov commented Nov 19, 2017

iamstarkov commented Nov 19, 2017

iamstarkov commented Nov 19, 2017

iamstarkov commented Nov 19, 2017

iamstarkov commented Nov 19, 2017 •

edited

Loading

paulmelnikow commented Nov 21, 2017

iamstarkov commented Nov 21, 2017

iamstarkov commented Nov 21, 2017

iamstarkov commented Nov 21, 2017

paulmelnikow commented Nov 21, 2017

[npm node] support for custom npm registry #1277

[npm node] support for custom npm registry #1277

Conversation

iamstarkov commented Nov 18, 2017 • edited Loading

paulmelnikow commented Nov 18, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

iamstarkov commented Nov 18, 2017 • edited Loading

iamstarkov commented Nov 18, 2017

paulmelnikow commented Nov 18, 2017

iamstarkov commented Nov 19, 2017

iamstarkov commented Nov 19, 2017

iamstarkov commented Nov 19, 2017

iamstarkov commented Nov 19, 2017 • edited Loading

paulmelnikow commented Nov 19, 2017

iamstarkov commented Nov 19, 2017

iamstarkov commented Nov 19, 2017

iamstarkov commented Nov 19, 2017

iamstarkov commented Nov 19, 2017

iamstarkov commented Nov 19, 2017

iamstarkov commented Nov 19, 2017

iamstarkov commented Nov 19, 2017

iamstarkov commented Nov 19, 2017 • edited Loading

paulmelnikow commented Nov 21, 2017

iamstarkov commented Nov 21, 2017

iamstarkov commented Nov 21, 2017

iamstarkov commented Nov 21, 2017

paulmelnikow commented Nov 21, 2017

iamstarkov commented Nov 18, 2017 •

edited

Loading

iamstarkov commented Nov 18, 2017 •

edited

Loading

iamstarkov commented Nov 19, 2017 •

edited

Loading

iamstarkov commented Nov 19, 2017 •

edited

Loading