CI Publish with Two-Factor Auth for Teams

[wesleytodd]

All package publishes should be done with 2FA enabled. This requirements means that publishing from CI is difficult because it requires the CI to have a OTP from the user. In order to maintain the 2FA you need some third party setup (a api to orchestrate the 2FA). @dominykas and the team at Near Form have done some exploratory work on this with Optic. This situation is even more complicated for teams managing publish access (like we want for Express).

The POC solves well the CI portion for a single maintainer, but it requires a infrastructure which I am not sure is best long term for OSS projects. It uses web push, so requires a Firebase account. Any infrastructure requirement is complicated for typical OSS maintainers. This is also vendor lock in.

In looking for options for a setup in Express, @dominykas and I hoped on a call to discuss. The outcome from that call was that we identified one clear best case scenario:

Npm builds a release manager which supports 2FA. So the idea would be that when CI pushed a release, it would be held in a pending state if it requires 2FA but no OTP was provided. You could then visit the website to see a list of pending releases. There you provide your 2FA OTP upon "approving" the release.

I am sure there are design and technical considerations to this approach, but we thought we should bring it up here and see what people think about this proposed solution. There are other approaches we discussed, and if I have time I will follow up with descriptions of those, but I wanted to get this posted to open up the conversation asap.

[ljharb]

I’m skeptical overall of having “not a human” do the one irrevocable thing, publish. imo the best practice is to have a human do the publishing, as the final defense against misclassified semver or other publishing bugs.

[wesleytodd]

I fully understand the concern. I personally am not just talking about automated releases necessarily. This problem even exists if the situation where a human clicks a button to run a CI job. On express we have talked about this as a possible way to have more publishers without compromising security.

final defense against misclassified semver or other publishing bugs.

Very valid concern, that said not everyone is as careful as you or thinks that is a humans job (semantic release for example). I think there is value in having a good story around managing automated or CI releases as a way to lessen the load on busy maintainers. It won't be right for everyone, but for those who need it it can be super helpful.

All this said, the original concern was how teams can manage publishing access when all it takes is one compromised publisher to take down the who ecosystem. If there are other approaches to this, I would love to hear ideas of how we can manage this risk.

[ljharb]

Compromising the CI publish token results in the same risk, so i think that the approach is unrelated do the original concern.

Perhaps npm could build a system where packages with multiple owners had to have multiple owners sign off on a pending publish, pursuant to rules configurable per package (and visible to users) - but i can’t think of anything “not npm” can really do at this point to mitigate the risk.

[wesleytodd]

Compromising the CI publish token results in the same risk, so i think that the approach is unrelated do the original concern.

Yes but if you build it properly you can prevent things like npm i from running on a box with a npm publish token. Doing this for all maintainers is more difficult than for one CI job.

but i can’t think of anything “not npm” can really do at this point to mitigate the risk.

We had a few ideas on this around using an intermediary service which held publish keys and was hooked into the publish flow. See the project Optic linked above. But I agree that npm is the best place to solve this problem.

[Eomm]

I think 2FA it is a matter of security where you have doubled the things a thief must steal.
So if you maintain these two things separated you could think to avoid the "dead man's switch".
Anyway, I like to have control of the publish, but maybe I push the publish button not so often.

[fun note] I built up some workarounds for this purpose with a bunch of server in different countries a many firewall (I was constrained 🤣):

const twoFactor = require('node-2fa')
var newOtp = twoFactor.generateToken('MY BOSS TOTP TOKEN')
console.log( { newOtp } ); // use the OTP in your CI publish

[mhdawson]

@darcyclarke @ahmadnassri, has there been any discussion within npm on this topic that you could share?

[wesleytodd]

I heard on the twitters that some sort of staged release was on the roadmap at one point 🤞

[dominykas]

@dominykas to write up things we can do with and without npm's involvement.

[darcyclarke]

@mhdawson @wesleytodd Apologize for the delayed response here.

I'll see if I can't address a few of the points made in the initial issue:

1. publishing from CI is difficult

• Understandable & unfortunate; I wish we had a better story around this today.

2. [paraphrased scenario] npm builds a release manager which supports 2FA

• I poked around internally and, unfortunately, it doesn't sound like this is something we have on our immediate radar. You're more likely to find that we'll ship org-wide 2FA enforcement before setting our eyes on a release manager with push to release.

3. All package publishes should be done with 2FA enabled.

• Totally agree and you probably already know that npm supports enforcing 2FA for publishing individual packages (image attached). That said, only 0.6% of all packages and 6.89% of all maintainers have 2FA turned on respectively (definitely metrics we'd love to try and improve).

Note: for any net-new capabilities we ideally try to surface meaningful data backing up the need/impact. We definitely want to support maintainers as best we can but if we were able to find data points around potential usage it would help with prioritizing this work.

---

Other Options...

In terms of supporting your efforts as they stand today, I think it might be best to avoid creating/maintaining a proxy OTP generator or publishing service; Instead, create a CI-specific user account and add them as a collaborator/author to your team/project. Then, generate an npm auth token that you pin to/use with your CI/CD environment (ex. GitHub Actions).

Definitely let me know if you think I've missed the mark or sentiment of this thread at all but hopefully some of the above is helpful insight/guidance.

[lholmquist]

Instead, create a CI-specific user account and add them as a collaborator/author to your team/project. Then, generate an npm auth token that you pin to/use with your CI/CD environment (ex. GitHub Actions).

I think this is a solid approach. It is similar to the concpet of a ServiceAccount in the Openshift/Kubernetes world

[bnb]

Totally agree and you probably already know that npm supports enforcing 2FA for publishing individual packages (image attached). That said, only 0.6% of all packages and 6.89% of all maintainers have 2FA turned on respectively (definitely metrics we'd love to try and improve).

@darcyclarke For what it's worth, most maintainers I've talked to have very consistently asserted that the reason they don't enable 2FA on publish is because the lack of a mechanism to actually publish following the current best practices that enables them to manually publish outside of having to run npm publish locally.

Unfortunately, until there's actually a mechanism to securely publish from a build, I don't think these numbers are going to change substantially.

[MarshallOfSound]

Hey @darcyclarke

In terms of supporting your efforts as they stand today, I think it might be best to avoid creating/maintaining a proxy OTP generator or publishing service

So fun story, over at @electron we've already built one of these, and been using it for a long time now. All of the npm packages that @electron publish are published using 2FA and using a system we built called CFA (Continuous Factor Authentication, I know, I suck at naming things).

One quick thing, it is still true 2FA - Only humans have the 2FA generator

We're currently in the process of updating bits of it to be more publicly accessible so that others can benefit off of the same system we do for safer automated package publishing.

Happy to share the system we use with you in its current state so you can see what we do and how it works. Also happy to chat the problems this solved and why we use it 😄

[darcyclarke]

@bnb I appreciate the feedback!

@darcyclarke For what it's worth, most maintainers I've talked to have very consistently asserted that the reason they don't enable 2FA on publish is because the lack of a mechanism to actually publish following the current best practices that enables them to manually publish outside of having to run npm publish locally.

I'm not sure I know what "current best practices" refers to; Also, I'd love if you could connect me with the folks that have given you this feedback so I can dig in further and/or surface how other people/orgs have solved this problem in the past.

Unfortunately, until there's actually a mechanism to securely publish from a build, I don't think these numbers are going to change substantially.

Again, there is already ways to "securely" publish (ie. auth tokens and/or auth tokens + CI-specific user account). For 2FA specifically though, you can run a CLI 2FA tool, such as krypt.co, to ease this pain.

Ultimately, I do agree that we should and probably will, at some point in the future, provide a means of doing "release management"; Which would hopefully address these concerns and make developer's lives easier.

@MarshallOfSound sounds awesome! I'm wondering if that has any overlap with krypt.co (which is what we use here at npm & covers A LOT of the concerns with 2FA management). In terms of release management though, I'd love to see how you're handling that. Definitely circle back if/when you go public with that work.

[dominykas]

Again, there is already ways to securely publish (ie. auth tokens and/or auth tokens + CI-specific user account).

Even if you have a CI-specific user account, you're still vulnerable, as if the token is stolen - people can publish all your packages. Maintaining separate accounts for each package is a hassle which I doubt anyone does. CIDR restricted tokens are a hassle because a) you can only set an IP range (not a DNS mapping) b) you cannot update the IP range (IIRC - was not able to create a new token just now with EBADKEY for whatever reason) c) you cannot even have a note on where you're meant to use the specific token.

These can be addressed, I'll admit, however that still means you're vulnerable, as Travis is defacto the standard (for now, anyways) and having Travis cidr range allowed for a token is as good as not having anything at all.

I would personally recommend that people do not use this approach, but rather publish from their own machines, with 2FA enabled.

I've also heard fears that people do not want to publish without 2FA, as there are (were?) plans to expose the information if the package was published with/without 2FA, meaning that whoever does not use 2FA will get a penalty of sorts (or at the very least they will be badmouthed on Twitter).

For 2FA specifically though, you can run a CLI 2FA tool, such as krypt.co, to ease this pain.

This looks quite promising and I will look into it - this might just be exactly what I was looking for with Optic. Thanks for the tip!

If it is what I think it is, then it's just the "team" aspect of it all that needs to be solved, and I have plenty of ideas how to tackle that.

[dominykas]

@darcyclarke are you in a position to share a bit more details about how you use krypt.co for 2FA?

It does not seem to provide a way to get an OTP remotely - it seems to only support U2F (which npm does not support)?

It seems that it can sign git commits, so I assume it can also use the same key to encrypt/decrypt stuff as well, but it's not very clear from documentation how to do it (side note: curious how that works - it could be quite some data to pass around over the wire to the phone and back?). So in theory it should be possible to have the encrypted OTP seed in CI, decrypt that via krypton with approval from the phone. Might be a hassle to set up, but I suppose that bit can be automated.

[ljharb]

What would be ideal for me is if you could autopublish from CI a tarball but it’d be marked “pending” for some period of time with a bot token, and then owners could make that live within that period via their own credentials, and it’d be as if they published it locally (it could expire after the time period). That way the build can be reproducible and verifiable, but a human’s still pulling the trigger.

It wouldn’t solve semantic-release entirely, but it’d get a lot closer without sacrificing as much 2fa safety.

[dominykas]

It wouldn’t solve semantic-release entirely, but it’d get a lot closer without sacrificing as much 2fa safety.

I've played around with a similar approach - I used semantic-release to push a tarball back into GH releases (example in assets), and then you can npm publish [tarball URL] (linked directly to the asset download) from your local machine (which will dutifully ask for OTP).

It's not a bad approach - I've considered building a tool that scans all your repos for unpublished tarballs in releases, etc - it could be very nicely automated. But before I go there - I'd like to see if there's a simpler, more usable way :)

[dominykas]

Partly note to self, but 2FA publish info is now public: https://blog.npmjs.org/post/188234999089/new-security-insights-api-sneak-peek

[ljharb]

Looking forward to a tool that can validate this publishing info from my full dep graph against a schema, like licensee can for license compliance!

[andywer]

Hey there! Sorry for being late to the party...

How about an alternative approach that is a bit less elegant, but works for a broader spectrum of use cases?

In CI you use a small CLI tool or maybe even just curl to query a service and request the 2FA code from the maintainer, authenticated of course. The maintainer receives an email, some push notification, ... with a link to the service's dashboard where they see a prompt for the 2FA code or whatever the other end requests. Maintainer enters the 2FA code, CI is still patiently waiting, receives the code now and passes it to npm.

Setup would be fairly straight forward: Sign up at service, create API auth token, add auth token to CI as env var and change npm publish to npm publish --otp=$(newtool request "npm 2FA code").

I built an early PoC a few weeks ago, but for a different use case (wanted to build electron app's binaries in CI, but without storing the production certificates and passwords in CI). Asymmetric encryption for end-to-end encrypted communication between the CI and the maintainer's browser is still missing right now, as well as authentication.

What do you think? It just occured to me that it might be a nice solution to npm publish in CI. Also happy to move the discussion to gitter.im or so, I don't wanna spam the thread – just wanted to share the idea.

[dominykas]

@andywer this idea is not a bad idea and it is covered in #282 (the PR listing options) and also some links to PoCs.

[terinjokes]

I prefer to leave npm logged out on my local machine, relying (almost) entirely on external tooling to publish artifacts. While it doesn't prevent tokens from being stolen, it does prevent a lot of cases of worm-like behavior.

I'd like to see more support for managing token lifecycles on npmjs.com:

• comments or tags to document usage, and support filtering of large amounts of tokens.
• changing CIDRs
• automatic expiration of tokens after a specific amount of time
• require 2FA with npm-login tokens, but allow publish tokens to bypass 2FA.

I'd be satisfied with an alternative approach, such as a pending queue.

[dominykas]

Just noticed that https://github.com/continuousauth is now available! Fantastic news and great work @MarshallOfSound et al.

[wesleytodd]

Another approach: https://github.com/erezrokah/2fa-with-slack-action