MFA and API Keys in CI/CD

[iMacTia]

I'm having issues with Rubygems MFA.
I've previously generated an API Key that I've been using in my CI/CD pipelines to publish new versions of gems.

However, after recently enabling 2FA/MFA, the pipeline has started failing in Travis with the following error:
Access Denied. Please sign up for an account at https://rubygems.org (Gems::GemError)

Although I completely agree with 2FA and MFA, I feel like the current implementation doesn't take into consideration automation and CI/CD Pipelines.
In my experience with other services, 2FA and MFA usually do not apply to API Keys. i.e. when using an API Key to authenticate (which is supposed to be long and hard to hack), you can bypass the 2FA/MFA check.

The current options, from what I read in the doc, are the following, but both of them are not good in the case of CI/CD Pipelines:

1. Disable 2FA/MFA checks for API and leave it on for UI only: this is of course insecure and nullifies the extra security as it seems like the CLI uses the API and you can authenticate to the CLI using less secure means other than API Keys (see below for more information).
2. Use the command line --totp argument: this works with CLI, but unfortunately is not possible to the top of my knowledge in CI/CD Pipelines, as I have no idea on how to make them sync with the Authenticator App on my phone.

The confusion on point 1 comes with the CLI.
On my laptop, the first time I used the cli I remember my username and password were asked:

After creating the account, use your email and password when pushing the gem. (RubyGems saves the credentials in ~/.gem/credentials for you so you only need to log in once.)

Now, that is of course unsafe, so I'd ideally like the CLI to request the MFA code in order to do anything. However, according to the documentation, that is only possible by enabling MFA for both UI and API (https://guides.rubygems.org/using-mfa-in-command-line/).

My suggestion would be to provide an extra option so that 2FA/MFA can be enabled for BOTH API and UI, but ignored when API authentication is done through API Keys only.

Any help would be greatly appreciated!

This issue is related to:

• Network problems
• Installing a library
• Publishing a library
• The command line gem
• Other

[mislav]

The way I understand it: "UI" = Rubygems.org web; "API" = gem push + other programmatic usage.

Even though gem push offers to authenticate using email + password, it converts that to an API key which it caches; see ~/.gem/credentials.

Personally, I do like to have both UI & API protected by MFA because I wouldn't like that an attacker who gains my password be able push new versions of my gems using gem push.

However, the downside is that I can't publish new gems from automation (CI/CD) anymore. I'm not sure what the correct soluton would be. Perhaps there could be a special API key that's exempt from MFA? I don't know if offering that would defeat the whole point of MFA.

[iMacTia]

Gentle bump on this one. Still an issue for me. Is there a working solution for this now?

[simi]

In my experience with other services, 2FA and MFA usually do not apply to API Keys. i.e. when using an API Key to authenticate (which is supposed to be long and hard to hack), you can bypass the 2FA/MFA check.

Can you point me to some docs another service provide for this? We can take a look and inspire in there. Also what do you suggest, to implement at rubygems.org side?

Is there a working solution for this now?

Would it be possible to create special CI user at rubygems.org having MFA disabled, add it as an owner to your gem, ensure that account is extra secured on your side (even without MFA) and use that on CI/CD for pushes? That way your personal account could stay with MFA, but you should be able to push on CI/CD on alternative account.

PS: I think we should move this issue to rubygems.org repo.

[iMacTia]

@simi @deivid-rodriguez the standard in this situation is to allow users to generate "application tokens". These are longer passwords supposed to be used with specific applications.
i.e. if I authenticate via the CLI I can provide my username and a generated token in place of the password. When that happens, the MFA code is not requested. If you instead provide the standard password, the MFA code will be necessary.

Top of my mind, both GitHub and Google implement this, although I'm sure I've seen this same practice quite a lot.
GitHub calls them "Personal Access Tokens" and has already deprecated standard password authentication: https://docs.github.com/en/rest/overview/other-authentication-methods#working-with-two-factor-authentication

[simi]

So for now it should be possible to temporarily simulate "application token" with new user with disaled MFA just for this purpose, right?

[iMacTia]

Yes correct, I can potentially create a new user with a really long password and MFA disabled, and use that one to push gems. I won't do that because my password is already reasonably long so I'm temporarily OK with MFA disabled until this is solved.
Managing multiple users is definitely more painful and not a long term solution, but at least you could suggest this solution for the time being (or increase password size).

BTW this kind of solution might work for Rubygems but it's worth nothing that it would not normally work for others (e.g. GitHub) when there are organisational constraints. For example, on GitHub, I could enforce everyone in my organisation to have MFA enabled, and this would have to apply to all users, including the dummy one you suggest to create.

If Rubygems were to introduce a security feature like "only allow MFA-protected owners to publish", then your solution wouldn't work anymore.

[deivid-rodriguez]

@iMacTia Yeah, I understand this issue and I agree it would be useful to have something like github's "Personal Access Token"'s in rubygems.org to make it possible to get both MFA and gem push automation at the same time.

However, I don't think exempting current API keys from MFA is a good solution since API keys currently give you access to essentially every API operation (API keys were recently scoped by gem operation, but even with that, an API key with the "gem push" scope still gives you access to pushing any gem owned by the key owner). Doing that would be essentially be removing the feature of protecting API keys with MFA.

I think we should only allow a key to be exempted from MFA if it's scoped by both gem and gem operation. So if you generate a "personal access token" that only allows to gem push new versions of gem foo, then it would be fine that this kind of key is excluded from MFA.

I'll transfer this ticket to rubygems.org because I also think it belongs there, and also to get feedback from rubygems.org folks.

[iMacTia]

Thank you @deivid-rodriguez and yes, I completely agree! GitHub's personal access tokens are basically Oauth tokens with different scopes that can be turned on and off at creation time. Ideally it would be great to have a similar mechanism with Rubygems, so that I can generate a token that only allows to push but not, for example to delete gems 👍

[sonalkr132]

personally, I won't recommend pushing gems from CI, especially third party CI. it is not worth the security risk. perhaps you can store the gem artifact somewhere from ci, validate the artifact then push it from your local workstation.

we should only allow a key to be exempted from MFA if it's scoped by both gem and gem operation.

a more granular scope is definitely required before we can consider disabling MFA. we are working on adding action scopes (#1962). I suppose we can build on it and further limit the scope by gem and client IP (something like cloudflare v2 API keys).

[iMacTia]

@sonalkr132 for personal projects that perfectly works, but for projects with many collaborators you don't really want to give everyone ownership of the gem, so the CI solution allows to give permission for doing different things (e.g. pushing new versions) to team members (e.g. anyone with commit permissions to master) while remaining the only owner of the gem.

I may be missing some security risks you mentioned, but the only alternative at the moment to have more collaborators is to make them gem owners as well, which arguably creates more risk.

I'm obviously not saying this is the best or only solution, and I'm open to alternatives from the Rubygems core team.
I just wanted to highlight that since Oauth tokens were introduced this basically became the standard for this sort of stuff.

[sonalkr132]

I may be missing some security risks you mentioned

I am primarily concerned about giving a third party (CI provider) permanent access to push to gems. The risk is of adding an entity in your "supply chain" which can potentially be avoided. Another obvious downside is that we can't use 2fa in CI.

with many collaborators you don't really want to give everyone ownership of the gem, so the CI solution allows to give permission for doing different things

is your concern about owners yanking versions? I would be interested in knowing why this workflow works best for you. Would a team account (#2258) where members have varying levels of access better handle your usecase?

Oauth tokens were introduced this basically became the standard for this sort of stuff.

I agree Oauth is more secure than API keys (even with scopes) just because it has a standard. Ideally, our API (and client) would support both methods. As it stands now our API key implementation is a bit lacking and we are working improving that first.
I would like to point out that Oauth is not a replacement for 2fa or a security equivalence. Even when we supported oauth, we would add 2fa to critical operations.

[sonalkr132]

I'd ideally like the CLI to request the MFA code in order to do anything... My suggestion would be to provide an extra option so that 2FA/MFA can be enabled for BOTH API and UI, but ignored when API authentication is done through API Keys only.

Please correct me if I am wrong, but it seems like you believe that gem cli doesn't use API key. Our CLI asks for user/pass when it couldn't find API key in the env, otherwise it uses API key for auth interactions (except gem sigin).
If you would prefer that we don't ask you for OTP on API operations (which included gem push), you can set your MFA level to UI only. If there is some clarification we could add on our guides, please point it out.

In my experience with other services, 2FA and MFA usually do not apply to API Keys. i.e. when using an API Key to authenticate (which is supposed to be long and hard to hack), you can bypass the 2FA/MFA check.
if I authenticate via the CLI I can provide my username and a generated token in place of the password. When that happens, the MFA code is not requested .. both GitHub and Google implement this

To clarify 2FA doesn't protect against brute force only. It could be possible that your API key was leaked and 2FA would protect you there as well. As of why Github doesn't ask for OTP when you have Oauth enabled, is beyond me. RFC for Oauth mentions that 2fa can be used as an extension. I suppose they are assuming that 2FA interaction won't be possible for API calls (IMHO, is it doable if you have a cli), perhaps @mislav have more insights here.

[iMacTia]

is your concern about owners yanking versions?

Primarily yes, or even worse they could potentially remove you as an owner. And I'm not of course saying they'd do this intentionally, but for example as a result of their account being hacked.
Secondarily, but appreciate this is completely unrelated from security, it allows you to setup a fully-automated pipeline for your gem (e.g. listing -> unit tests -> integration tests -> pushing).
Advanced team permissions in Rubygems would solve the security concern but still prevent to reach full automation.

If you would prefer that we don't ask you for OTP on API operations (which included gem push), you can set your MFA level to UI only. If there is some clarification we could add on our guides, please point it out.

That's not how I understand this works. My understanding is that "UI Only" covers only the website. If I login using user/pass via CLI, no MFA will be asked unless that's is enabled for "API" as well, which is the main issue for me. Ideally it would be better if the distinction was not on UI/API but rather User-pass / API Token.

To clarify 2FA doesn't protect against brute force only

Absolutely. There's no doubt that having 2FA enabled on top of API keys is indeed safer than not having it.
That said, it is a wide-spread convention to have the possibility to enable it only for user/pass authentication and not for API keys/tokens (as we mentioned the example of Google and GitHub). This allows us to achieve automation in our CI/CD pipelines while keeping security under control. I'm far from being a security expert so I'm unable to justify why these big companies consider it safe, but my guess is that a very long, scoped API token (similar to Oauth, so with limited powers) is considered safe enough not to require MFA, while your user/pass credentials are not.
For example, even if my push-only token ended up being exposed, that would only allow for pushes to happen. I'd be immediately notified by Rubygems, at which point I could login, invalidate the token and yank the wrong version.

[sonalkr132]

Ideally it would be better if the distinction was not on UI/API but rather User-pass / API Token.

Thank you for clarifying this. I think it makes sense to have this distinction over what we have as of now.

it is a wide-spread convention to have the possibility to enable it only for user/pass authentication and not for API keys/tokens
scoped API token (similar to Oauth, so with limited powers) is considered safe enough not to require MFA, while your user/pass credentials are not.

sorry, but I am not sure that MFA is not asked for them because Oauth is considered safe enough. Oauth and 2fa have very different security guarantees. In the general sense, API endpoints are not meant to be used by humans, which ends meaning that providing or asking for OTP just doesn't work (for most use cases).
Equivalence of our workflow/2fa implementation could be what npm does, where it has similar MFA levels, Package delivery is more sensitive than generic API usage. Our recommended method should remain that we want all users to use MFA for publishing gems.

Advanced team permissions in Rubygems would solve the security concern but still prevent to reach full automation.

Please leave your comments/use case on the issue I linked if team account solves some of your problems.
As for automating the flow of publishing gems, I think if you prefer the convenience of publishing gems from CI, you will have to compromise with security and set your MFA level to UI only. Please feel free to send a PR to update this option to include the endpoint used by gem signin. Thank you for bringing attention to this.

[iMacTia]

Thank you for clarifying this. I think it makes sense to have this distinction over what we have as of now.

That's all I was asking with this issue, having that distinction would make a lot more sense in my opinion.

[iMacTia]

@sonalkr132 congrats on releasing the API Key scopes, I just saw the change in Rubygems.org 👏
I was wondering if now it might be a good time to have another look at this?

[sonalkr132]

The issue we need to work-around for this is that we can't change the current behavior of "UI only" level. I think we have two options here:

• We can consider "UI and API" as the default behavior for all keys and give users an option to override it in API key form. ie if I check this box, don't ask OTP even if the MFA level for my account is "UI and API".
• We add a new MFA level "UI and gem signin" (or something similar), which would mean ask OTP for "UI only" + gem signin command.

The argument for using "UI only" wording (over login with user:pass) was that there are UI actions like changing password also ask for OTP even tho they don't ask you for user:pass.

What do you think? I guess the second option is more straightforward. This can get done much faster if you can put in the legwork for creating the PR for this. I would be happy to help and review. My todo is filled up until the end of Feb.

[iMacTia]

Thanks for the details @sonalkr132.
I personally like solution 1 more, as I believe it does a better job at separating the 2 concerns, but I think I understand why solution 2 is actually easier to implement. I'll see if I can spend some time on it and start a PR

[iMacTia]

@sonalkr132 there you go 🎉
#2601

[f-moya]

There might be a safer alternative http://tenderlovemaking.com/2021/10/26/publishing-gems-with-your-yubikey.html

[danielsilva]

I am not sure if this is working as intended. I am having issues trying to use "UI and gem signin" with an API Key that has MFA disabled.

This is my setting:

This is the API Key setting:

This is the message that I get when running GEM_HOST_API_KEY=*** bundle exec rake release:

Pushing gem to https://rubygems.org.../
Rubygem requires owners to enable MFA. You must enable MFA before pushing new version.
Error: Process completed with exit code 1.

I appreciate your help.

[iMacTia]

mmmh this was and still is working fine for me, although I haven't changed my token recently and I noticed a change to API keys has recently been introduced (Oct 31).
Maybe that is causing new tokens to not work anymore? Or maybe the steps to make this work are slightly different now?

[simi]

Hello @iMacTia. It seems this was regression recently introduced being fixed at #4272. I can ping you once this is deployed to confirm everything is ok again.

[iMacTia]

I haven't experienced any issue yet (maybe because my token was generated before the bug was introduced?), but I'm sure @danielsilva would like to know when the fix ships 😄

[simi]

@iMacTia just got shipped.

[danielsilva]

That was quick!
Thank you guys. I appreciate your work.