repo server app secret isolation for multi tenant

[btrepp]

Summary

There should be a way of providing secrets to the argocd repo server. That can't leak to other tenants in the server. Eg I shouldn't be able to use team As secrets from TeamB

Motivation

Argocd currently states that it's unopinionated about secrets. This is good. However it does seem some design decisions do push you down some compromises, or at least prevent you from using secrets during template expansion.

Many secret management systems require you to 'bootstrap' and initial secret. Eg a gpgkey or password. Most plugins in the argocd ecosystem currently achieve this by volume mounting the secret.

This is a problem as I don't believe the repo server really protects against anything being leaked. Especially as to work this way you are now running extra plugins.

An application could very easily ask for the secrets intended for another application. Also concerning is that this isn't super documented :).

I also think someone may be able to leak the bootstrap key or secret this way, from well intentioned plugins.

Proposal

At the minimum it would be nice if applications and app projects could have some way of allowing specific secret access for repo server.

Eg to cover my case. Having an app-project enforce 'allowed secrets' as a reference to an existing k8s secret, and extending the application to work with this too. That way Argo would only mount my gpg private key (for secret decryption) when it's used on that application.

Alternatively, perhaps spinning up a repo server per application, that mounts the app specific volumes could work too.

This capability would be nice, as it would be re-useable against multiple secret inflation systems, implemented as config plugins.

[crenshaw-dev]

I think we moved in the right direction by isolating the sidecar CMP filesystem from the repo-server and making manifest files very short-lived with randomized paths.

I'd love to keep Argo CD unopinionated about secrets. But I agree, addressing this common use case securely probably requires some opinions.

Secrets should probably only exist on disk during manifest generation, and they should probably be under randomized paths. A CMP author could implement that, but it's not really good to re-implement the feature for every CMP.

And agreed, secret access should probably be segmented on Project.

@jannfis has done some work pondering repo-server-per-tenant. Maybe he can share thoughts on that.

[btrepp]

At the moment, trying to integrate the secrets. Argocd has an opinion :), it's don't use argocd for secrets at all :). That being said I think it's important people can bring there own secret management, so the philosophy of not locking on in on one methodology is incredibly sound.

I was exploring implementing a CMP, I have a rough design of generating secrets in temp files, then using them as kustomize overlays. I have hit a roadblock as how to build a CMP isn't 100% clear, theres great docs on how to echo a shell command as a CMP, but where the working directory is, and the environment around it is unknown. However my CMP would need access to the app files. This has slowed me down a bit :).

Previous iteration I tried was a kustomize secret generator plugin. This works on dev machines, but kustomize best practice is to implement them as docker containers, this isn't currently supported by argo, and may not be as it's fairly complicated fairly quickly.

Would be great to have some guidance :)

[crenshaw-dev]

Argocd has an opinion :), it's don't use argocd for secrets at all :)

Yep. Which sometimes ends up pushing folks towards risky hacks. :-(

I was exploring implementing a CMP, I have a rough design of generating secrets in temp files, then using them as kustomize overlays.

Sounds interesting! I'll be curious to see how you protect those (temporary?) secret files from problems like directory traversal. Are you using a sidecar CMP or an argocd-cd CMP? The info below will relate to a sidecar CMP because I highly recommend going that route.

but where the working directory is, and the environment around it is unknown

The working directory is the full app source path (i.e. the root of the repo plus the path from the application spec).

The root of the repo is copied into a temporary directory with a random name immediately before manifest generation and deleted immediately after manifest generation.

kustomize best practice is to implement them as docker containers

Yeah, full containerization is definitely best when dealing with secrets. @leoluz investigated creating functional containers for CMPs, but it looks like cloud providers don't consistently support Linux user namespaces yet. So for the moment we're relying on filesystem separation (by using a sidecar with minimal shared volumes) and ephemeral repo contents.

[btrepp]

Sounds interesting! I'll be curious to see how you protect those (temporary?) secret files from problems like directory traversal. Are you using a sidecar CMP or an argocd-cd CMP? The info below will relate to a sidecar CMP because I highly recommend going that route.

The main idea was temporary files and deleting on success or error. This was going to be in a sidecar CMP, but I am struggling a little to write a sidecar :)

Another part was to run GPG in a different container, using the remote socket for the agent (this is nice so that you can use a key, but not actually access it). I'm not sure how the GPG agent is running in argo, but might be a secondary improvement. Long term I want to try to actually use USB tokens for this, but it's a seperate problem to solve :).

The root of the repo is copied into a temporary directory with a random name immediately before manifest generation and deleted immediately after manifest generation.

That's good to know. It can probably simplify how the kustomize approach may work. I was attempting to manage the temp dir myself.

Unfortunately I can only really think of two simple implementations.
Above was a bit of a 'kustomize wrapper', it generates secret templates and calls kustomize. This seems to be really against the kustomize philosophy, though annoyingly no best practice is offered. I can tell as a tool it's in folder thinking mode.

With the above, the most likely easy way forward is a second repo eg actual_app_plus_secrets, secured by SSH keys, containing the secrets (in clear text :() and then using your bases on your 'normal' repo. It feels wrong to store clear text secrets in git though, even if it is technically in a 'private' repository.

I think for plugin authors, given the various tools (passwordstore, sops, git-crypt) the missing argocd feature would be management of a GPG key. If repo server could be initialised with a secure managed private key, then these tools can either decrypt what they need.

E.g on app project, something like

  signatureKeys:
  - keyID: 123ABCD
  projectKeys:
  - keyID: ABCD123

Which would mean argo is allowing that private key to be used in to repo server for applications in that project. It would be added to the GPG agent only for that project, and not accessible to other projects.
This would go a long way on the risky hacks. A lot of the samples I have seen just mount the private gpg key in a volume mount, then load it in to GPG. Which means it's sort of always there, and would suffer from directory traversal.

This could also probably be fine if argocd generated the key if not present, though that might be difficult. If generated, anyone wishing to send secrets to argo, would using the public key to encrypt, and if it does change, then you rotate in your tool.

[crenshaw-dev]

That would put Argo CD squarely in the "having an opinion about secrets" space, but maybe that's fine. I'd be very interested to see a proposal document based on this idea, but I realize it's a big time commitment. Regardless I'll mention this idea in the next security meeting.

[btrepp]

Thanks :). To help keep it unopinionated, I am not asking for anything over a 'project identity'. If argocd has any way of identifying different applications/projects via the git repo, this would let plugin authors build there auth plugins of this 'master id'. This isn't too different to the fact that there is a 'repo clone identity' so there is some precedence.

As GPG keys can be SSH keys, these could even be unified, though it's a bit tricky.

I've actually made the first pass of my plugin that utilises password store https://git.sr.ht/~btrepp/argo-pass-sidecar
For now this is giving up and considering argocd as asingle monolith, so no multi-tenant safety.

In doing this, I have learnt a few things about argo internals, and I think it would be possible to offfer this 'identity' via multiple gpg-server sidecar containers, and having the plugin select which socket to use via an env var.

So the concept is probably achievable out of core argocd code. There's a bit of complexity in using gpg to achieve this though. It is one of the most user unfriendly cli tools I have seen. (Literally cases were --daemon flag blocks the current terminal, which is... incredibly wrong ;))