vaults/secrets management

[jdx]

as the [env] section of config files will very likely contain secrets, there should be a way to manage/encrypt these. I have 2 rough ideas here:

1. support built-in encryption. Allow committing an encrypted config into a repo with a single env var or something used to decrypt it. Maybe there's opportunity for integrating with macos keychain or yubikeys or something like that.
2. support backends. I'm thinking things like hashicorp's vault, macos keychain, and maybe directly integrating with providers like heroku and vercel to pull the env vars from a remote application. Or maybe we do something like dotenv-vault.

Authentication, Authorization and Auditing

[jdx]

on a related note, I've also been thinking that I want a way to share my configs either with myself or others. Right now I'm putting it into my dotfiles repo, but is there an opportunity there to make sharing more of a built-in concept so one wouldn't need a dotfile repo? Also how could one share configs with their team when they don't go into a project?

[reitzig]

ad 2: FWIW, twpayne/chezmoi supports a range of password managers. I'm just starting off using both chezmoi and mise (👋), but I strongly suspect that there's considerable overlap in "adapter" code.

Personally, I would favor approach 2. Sharing files with references to secrets feels way safer than trusting Yet Another Tool™. Plus, not storing a copy (encrypted or not) makes rolling secrets a lot easier.

That said, a personal note on UX. I've written something like

export TOOL_USER="$(op get item "Tool Token" | jq -r '.details.fields[] | select(.name == "username") | .value')"

in an .envrc before, plus bashey sins to authenticate with op. First, if mise can handle that for me, great. Second, asking to unlock the vault when entering a directory can be disruptive; a separate, non-automatic mechanism for loading secrets may be more pragmatic than always resolving secrets.
So, in the end, I may just use op run and stick to plain-text references in env files. For 1Password at least, wrapping op may be beyond the point of diminishing returns.

[asmod3us]

I use sops with a direnv plugin. It really hits the smooth spot for me with a single yaml file, shell env vars and/or json/yaml output and using pgp keys. Would be keen to try it in mise!

[hahuang65]

Would like to make a suggestion. Some other tools, such as weechat or aerc, allow password and password_cmd as config options. The latter runs an arbitrary shell command.

This would allow users to use any arbitrary secret storage, provided it has a CLI.

Personally I use keybase for personal stuff, my work place uses Vault as well as AWS SSM.

This would be nice as it wouldn't limit what could be used, and puts the onus of security on the tool/user, rather than something mise would have to maintain.

[roele]

Is the usage of CLIs not already supported via tera templates?

Following assumes VAULT_TOKEN is set somehow already.

[env]
USR="{{exec(command='./vault kv get -address=http://localhost:8200 -mount test -field=usr myapp')}}"

[joshbode]

I'm working on a SOPS plugin for Mise, based heavily on asdf-sops with some stuff borrowed from the poetry plugin, also.

The initial version is here: mise-sops

I'm still playing around with it but so far it's working great for me - the caching is working so it's very fast once the file is loaded.

[tools]
sops =  { version = "3.8", filename = ".foo.env:.bar.env", names="FOO:BAR:BAZ" }
# names optional to filter exported variables

(tool values seem to need to be strings, otherwise I'd make filename a list rather than separating with : :)

All feedback is welcome!

[jdx]

hooks probably got us a bit further along and improved some of the related issues like hook-env not running on cd all the time.

Just dropping by to say I still want this. Dunno when I'll work on it.

[jdx]

ok here is my plan, but this is a huge undertaking to get everything I want I feel. I am very unfamiliar with these tools so tell me if I'm off-base or anything. I think this is going to be roughly done in several phases. First is to get encryption of mise.local.toml (it'd work with any config, but that's probably the one most likely to be used for this). Second would be secrets inside of mise.toml. Lastly, I see some potential for improving caching and security further.

Milestone 1 - encrypted configs

This is basically sops/rops integration. Currently, sops is lacking toml integration but it looks like people are at least poking at the problem.

So here's the workflow:

• user manually creates rops-encrypted mise.local.toml
• mise will automatically decrypt using rops-lib in rust when it encounters sops-encrypted configs

From the user's perspective, this is what you can expect:

$ age-keygen -o age.txt
$ Public key: age10v0k6lrd2hzjamlnatqzpcl7ky0n04f6vum9hvjxst0ce0nmsvdshjjly6
$ echo "env.SECRET = 'mysecret'" > mise.local.toml
$ rops encrypt --in-place --age age10v0k6lrd2hzjamlnatqzpcl7ky0n04f6vum9hvjxst0ce0nmsvdshjjly6 mise.local.toml
$ mise env | grep secret
export SECRET=mysecret

You can also then view its contents or edit it with rops (and later sops when toml support lands):

$ ROPS_AGE_KEY_FILE=age.txt rops edit mise.local.toml
<opens editor>
$ ROPS_AGE_KEY_FILE=age.txt rops decrypt mise.local.toml
env.SECRET = 'mysecret'

I think this is a good place to start. While you wouldn't be able to use something like 1password directly in mise.toml, you could write a script yourself that generates and then encrypts mise.local.toml so it would unblock people without needing to have raw secrets sitting in a project directory.

Milestone 2 - secrets in mise.toml

I think there are 2 ways I'd like to see secrets imported directly into mise.toml based on whether you need a single secret or want to import a set of them. For single secrets, this would use templates which would be usable in most places in mise, e.g.:

[env]
AWS_SECRET_ACCESS_KEY = "{{secret(run='op read op://dev/credentials/aws_secret_access_key')}}"

And we could add some syntax sugar around this for common secret providers (possibly defined via vfox plugins):

[env]
AWS_SECRET_ACCESS_KEY = "{{secret_op(read='op://dev/credentials/aws_secret_access_key')}}"

Note

#1912 is a prerequisite for being able to use op here if mise provides it itself, otherwise op wouldn't normally be available.

The second way would be when you want to fetch a group of values:

[env]
_.secret.json = "op read op://dev/env/dev"

Again with sugar:

[env]
_.secret.op = {json="op://dev/env/dev"}

However the latter use-case I'm a little less sure about, is anyone doing something they can give me an example of how they do something similar it would help me design it. Something like dotenv-vault pull. I also find the ergonomics around op a little wonky but maybe it's just me (haven't used it much). There seems to be 2 ways to fetch things that I don't understand the differences of: op read and op item get.

I think for the sugar stuff it would be good to get an idea of as many secret managers as possible and get some pseudo-code configs about how it could look.

Milestone 3 - caching

In order for milestone 2 to work well (really at all except testing), mise needs to cache the results. You won't need your secrets refreshed every time the prompt is displayed of course.

mise doesn't run a daemon. While I could store secrets in the environment for hook-env users, that means anyone not using mise activate wouldn't be able to use secrets. I think the only realistic solution is to store files containing raw secrets in an internal directory like ~/.local/state/mise/secrets.

We could ensure this is chmod 0600. I still don't love the idea of secrets sitting there though. To improve that, I think we should encrypt the cached files similarly to encrypting configs in milestone 1. The problem is that I won't necessarily have a key available to do the encryption—in milestone 1 the user needs to configure that for it to work. I don't think requiring config to use this is good enough. That said, if it were configured it might be like this:

export MISE_AGE_KEY_FILE=~/.age/key.txt

or with ssh:

export MISE_AGE_SSH_KEY=~/.ssh/id_ed25519

Under the hood, mise might use rops or rage for this—not sure. rops might provide useful features, and we'd already be using it. rage is already a dependency of rops so it could be used directly with an additional crate. It does not appear rops supports ssh (or it's automatically configured somehow) so rage might be necessary.

Lastly, if no age configuration is provided, mise could still default to encrypting using a symmetric key randomly generated in build.rs—making it unique every time mise is built. Of course precompiled binaries would be able to extract the same file, but at least it would make it harder to read secret cache files generated by mise since an attacker likely would have no clue which binary created the secret or maybe be unable to have access to it if the user ran cargo install mise to build from source. I assume it's hard but probably not impossible to fish the string out of the compiled binary.

Symmetric encryption is obviously not foolproof, but what we're talking about are just cache files for mise that only the current user should be able to access so they're already relatively secure, this would just make them a lot harder to decrypt if a user had access to them without the user needing to configure and figure out how to use age. Defaulting to ssh is probably a bad idea because if the ssh file is simply stored as raw text with no password it'd actually be worse than the symmetric generated key.

Paranoid mode would disable this feature and require users to configure age directly—maybe checking to ensure the ssh key is not a plain text file.

[andrewthauer]

In regards to 1password / op you should really be looking at secret references with op inject instead of op read imo. I can't confirm this atm, but I think op read would be much less efficient, since it would be a key by key call vs op inject which takes a file or stdin and outputs the entire file/input with resolved values. I much prefer this approach as it allows you to leverage using .env files.

For example:

Setup:

AWS_SECRET_ACCESS_KEY="op://dev/credentials/aws_secret_access_key"
# ...

Usage:

# output the resolved .env file
cat .env | op inject

# this will put the resolve .env into the current shell env
source <(cat .env | op inject)

# or use within a direnv .envrc file you can do to inject the resulting secrets into the env...
dotenv .env
direnv_load op run -- direnv dump

I can see benefit in encrypted values in the mise.toml, but I would mostly want to manage secrets outside of mise and just have a nice way mise can ensure variable secrets are in place for the current workspace to use. Possibly similar in concept to the direnv solution noted above. However, perhaps this is a different feature set / solution to this issue. What I'm describing is more related to #3542 I think. However, I think its relevant to discuss here as well as they are related imo.

[jdx]

I ended up implementing milestone 1 this a bit simpler than I described above. I went with just .env.json as an encrypted file that can be imported with _.file = ".env.json". If it's a sops file, it'll automatically get decrypted.

I think with that, and maybe a cd hook to create this file or something it should be a reasonable solution for right now.

[joshbode]

Thanks @jdx - I've tried it out and it worked well for me.

In fact, it almost entirely obviates the exec-env hook I created in https://github.com/mise-plugins/mise-sops, apart from not being able to selectively filter which secrets are loaded, though I think I can live with that, since I can load secrets from multiple files (last wins), e.g.

[env]
_.file = [".base.json", ".env.json"]

which means I can divide up the secrets across files so they're loaded selectively.

(also, I will need to stop using GPG with SOPS since I don't believe it's supported by ROPS, but that's no big deal)

[jdx]

there are probably going to be several gaps with using rops which could be fixed by shelling out to sops. I didn't want to do that for v1 since I think most sops users will probably be using age and I want it to be fast, but shelling out is totally feasible. idk what the performance impact will be, I suspect it'll be noticeable but probably not a dealbreaker.

[joshbode]

I think it's fine to just use ROPS with Age - you've made it straightforward to point to a key file, and GPG is... a whole other thing, and not hugely user-friendly (agents, trust stores, etc)

[jdx]

I added it anyhow—I know for sure someone will need this #3603

[jdx]

In fact, it almost entirely obviates the exec-env hook I created in https://github.com/mise-plugins/mise-sops, apart from not being able to selectively filter which secrets are loaded, though I think I can live with that, since I can load secrets from multiple files (last wins), e.g.

sorry for killing your work, but I assume you probably prefer it baked in anyhow—if not least for the performance. It wasn't for naught though, I used it as one of my references for how things should work.

[jdx]

In regards to 1password / op you should really be looking at secret references with op inject instead of op read imo. I can't confirm this atm, but I think op read would be much less efficient, since it would be a key by key call vs op inject which takes a file or stdin and outputs the entire file/input with resolved values. I much prefer this approach as it allows you to leverage using .env files.

This makes me wonder if we should have {{secret*}} templates at all or just promote the current workflow with sops as the way to do it, or just raw .env.json files outputted from a secret manager. You could even configure a mise hook to build it automatically and (optionally) encrypt it with sops:

[env]
_.file = ".env.json"
[hooks]
enter = '''
# of course instead you could output a normal .env file,
# but this way .env can be committed but .env.json can not be
cat .env | op inject --file-mode json > .env.json

# this step is purely optional, .env.json wouldn't be committed
# but it might be nice to just keep secrets out of your working directory anyhow
sops -i encrypt .env.json
'''

I'm wondering if we even need or want further syntax sugar on this. I am working on lazy env vars which can use tools—though the only use case I feel like I ever see around that is for secrets management. That road involves caching and a lot of complexity both in the codebase and for users to deal with.

I'm leaning towards promoting .env and .env.json as the way secrets are done in mise and leaving it at that. That said, we might be able to put some syntax sugar on the above like this which under the hood would do rougly the same thing as my first example config:

[env]
# this will call `op inject --file-mode json` and
# put the results in an encrypted file somewhere in mise's internal directories
# it'd save that for 8 hours (configurable of course),
# perhaps when entering the directory each time or maybe not
_.secrets.op = ".env"

[joshbode]

sorry for killing your work, but I assume you probably prefer it baked in anyhow—if not least for the performance. It wasn't for naught though, I used it as one of my references for how things should work.

No worries! Integrated is better, and once usage of the new feature becomes established, I'll probably deprecate the exec-env part from the plugin and it can just be used for installing SOPS releases.

The performance is definitely better, although it was mostly being incurred as a first load hit (upon cd into the project) because caching was occurring, I believe

[andrewthauer]

This is starting to look promising!

I tried the integrated SOPS solution and it works well. I personally haven't done much with SOPS, so I'm not sure how one would effectively use AGE within a team or org. Would it not be more common to use KMS or something for this? Any thoughts on if and how that might be supported?

As for non SOPs approaches (e.g. op), I think the core requirement comes down to something like the enter hook being able to source into the main shells environment. I believe if mise has a way to do this, then it should suffice and integrate with most tools. The key is not running on every prompt (only when entering a project area like direnv). I think even sops exec-env could work this way as well if you didn't want to use the integrated approach. Disclaimer ... I haven't actually tried to use hooks yet.

[jdx]

I tried the integrated SOPS solution and it works well. I personally haven't done much with SOPS, so I'm not sure how one would effectively use AGE within a team or org. Would it not be more common to use KMS or something for this? Any thoughts on if and how that might be supported?

I only started looking at these tools for this task so I'm not super familiar either. I think teams would probably use something like kms or vault (which they can with MISE_SOPS_ROPS=0), but age could definitely work. You'd just need to share a recipients file with everyone's public key in it—basically like we do with ssh public keys for ssh.

As for non SOPs approaches (e.g. op), I think the core requirement comes down to something like the enter hook being able to source into the main shells environment.

Can you elaborate? I'm not sure what you mean. Hooks normally execute as a bash subprocess. You'll have access to all the same env vars of course, but things like shell aliases will be missing. There is also the ability to run "shell hooks" which are a bit scary because they just execute raw shell code.

Of course we also have caching for template commands too, so there are multiple ways to solve this problem now.

I am not sure if this should be closed out or not. I think the only potential remaining thing would be the "sugar" I mentioned:

[env]
# this will call `op inject --file-mode json` and
# put the results in an encrypted file somewhere in mise's internal directories
# it'd save that for 8 hours (configurable of course),
# perhaps when entering the directory each time or maybe not
_.secrets.op = ".env"

anyone have thoughts on this? It certainly makes it more concise, but is it actually any much better than manually configuring a hook? I'm concerned something like this might result in me making endless patches for customization when people might be better served by a more verbose hook that doesn't require understanding what env._.secrets.op means.

[andrewthauer]

Can you elaborate? I'm not sure what you mean. Hooks normally execute as a bash subprocess. You'll have access to all the same env vars of course, but things like shell aliases will be missing. There is also the ability to run "shell hooks" which are a bit scary because they just execute raw shell code.

Yeah, I could being getting confused since I've only looked at the docs for hooks and not tried them. Doing something like the example below mostly works already for what we'd need, but has the side effect of running on each prompt. Which if the command is fast it's ok but op inject isn't the fastest so something that works like what I think "enter" does (or caches the results) would probably work:

# essentially something that does ...
# source <(secret-env-cmd)
# where `secret-env-cmd` provides a bunch of `export FOO=bar` lines
# the caveat is that `some-command` would only be executed when you cd into a directory that contains a `mise.toml` file
# ... if you cd into sub directories it doesn't re-run `secret-env-cmd`

[env]
_.source = { cmd: "some-command .env", some-mode-for-caching: "" }

Correct, me if I'm wrong, but the sourcing part is easy and perhaps the cd into other sub directories, but the leaving the parent directory containing the mise.toml and restoring the environment is the tricky part? I think this is more or less what direnv does essentially.

I think the only potential remaining thing would be the "sugar" I mentioned:

Could we not generalize _.secrets.op into something like this:

[env]
# essentially does what I described in the example above (with caching etc) but without the hook/cache mode
_.secrets = "op inject .env"

I think the caching of the decrypted files for a period of time could work, but I'd really rather prefer to never persist plain text secrets to disk anywhere. Currently when I do this with the source <(op inject .env) type patterns it's only in memory (as env vars) and mostly always up to date when it's paired with direnv to automate it.

[jdx]

converting to ideas for now until we have something a bit more concrete to put into a new issue