A potential risk in emissary-ingress can be leveraged to make a cluster-level privilege escalation

[younaman]

Summary

The emissary-ingress has two deployments in the namespace emissary. One of them is emissary-ingress, which has three pods running on worker nodes. These pods have cluster role emissary-ingress which has "get list watch" verb of "secret". Thus, if a malicious user controls one work node with the pod of deployment emissary-ingress, he/she can leverage the emissary-ingress deployment's pods to get the cluster admin's secret, making privilege escalation.

Details

In our local environment, we have a three-node cluster with Kubernetes v1.25, we deploy the emissary-ingress following its official guide(https://www.getambassador.io/docs/emissary/latest/tutorials/getting-started).
In our local environment:
The emissary-ingress deployment has three pods, one of them runs on worker A, and two of them run on worker B.
Thus, if a malicious user can access one worker node, he/she can leverage the related pods' token to get the cluster admin's secret, making a cluster-level privilege escalation.

Mitigation discussion:

Overall, the emissary-ingress should review its source code and restrain the secrets which its deployment can access. More specifically:

1. The emissary-ingress maintainer can use the "Resource Names" to restrain the name of secrets that can be accessed. However, it may need a careful review of the source code to mitigate the risks without disrupting its functionalities.
2. The emissary-ingress maintainer can use the role binding, not the cluster role binding, to bind the cluster role to the emissary-ingress deployment. Thus, he/she can only use the token to get the secret in the "emissary" namespace and mitigate the potential risks of getting the cluster's admin token.

A few questions:

1. Is it a real issue in emissary-ingress?
2. If it's a real issue, can emissary-ingress mitigate the risks following my suggestions discussed in the "mitigation discussion"?

Impact

It is a privilege escalation, a malicious user can get the cluster admin's secret and do whatever he/she likes to other users' workloads.

[younaman]

Besides, I have checked the private security policy of the emissary-ingress and the official website of the emissary-ingress, and I don't find any mailing list to report this kind of problem. So I write a "security issue" by opening a public-accessible GitHub issue. I am sorry if it caused ethical considerations.

[cindymullins-dw]

Hi @younaman , thanks for raising these concerns. If you'd like to contribute or discuss these concerns further we do have a monthly contributor's meeting you're welcome to join.

[cyrus-mc]

@younaman

What exactly do you mean get access to a worker node? If an attacker can get access to the underlying host they can do a heck of a lot more than just use the Emissary permissions to read secrets within the cluster. So while you could say that maybe the RBAC rules should be tightened if possible, I don't think this is really a privilege escalation against Emissary (there are plenty of other components out there that have cluster roles giving ability to read secrets) as compromising a worker node pretty much opens you up to privilege escalation through many vectors.

[younaman]

hello@cindymullins-dw, It's a pleasure to receive your invitation. By the way, can you ensure that the potential risk is a real issue? And could other contributors give me some more detailed suggestions? Besides, in monthly contributor's meeting you mentioned, I find there is a weekly troubleshooting, can we discuss this potential risk in detail in the weekly session and find out some mitigations?

I am looking forward to hearing back from you.

[younaman]

@cyrus-mc @cindymullins-dw Knock knock! Are there any updates?

[cindymullins-dw]

Hi @younaman , would you like to join the next weekly Emissary session to discuss? That would be on Thursday at 2:30pm ET. You can use the link you posted to join us.

[zolaer9527]

@cindymullins-dw a weekly troubleshooting,is a Invalid conference number. Is there any update adout the link of weekly troubleshooting in document? And the conference number is invalid, how can i join the meeting on Thursday at 2:30pm ET?

[jasonstitt]

Wiz is also flagging this for us as an issue. Note, the escalation path here isn't node compromise -> secret compromise as that would not involve emissary per se. The escalation path is emissary compromise -> secret compromise, and it's specifically imporant because emissary is an exposed service that, since it may handle most of an org's HTTP traffic, has a large exposed surface.

What's unclear to me is why emissary needs read access to all secrets. I see based on an old issue/PR (#1365) that this was considered optional at some point. But I haven't found documentation on what the secret watch is for (nor is AMBASSADOR_NO_SECRETS in the documentation and it appears it may have been removed from code since then).

[younaman]

@jasonstitt Thanks a lot! I am glad that this issue will still be noticed by someone after these days. So you mean that the secrets have been removed and this problem has been fixed secretly?

[jasonstitt]

No that was just some initial research.

[bsamsom]

Does anyone know what secrets it actaully needs permissions for?

I couldnt find what it was looking for in the codebase, and the only secret i could find thats related was the emissary-ingress-webhook-ca secret the emissary crds add.

Ive also been looking at locking our emissary down and as mentioined above the AMBASSADOR_NO_SECRETS flag looks to be gone with no new replacement.

I tried setting:

rbac:
  create: false

In my helm chart and deployed my own Role && RoleBinding in hopes that error logs would indicate what secrets I need to add in resourceNames

but no luck, the erros logs when emissary cant acces secret just look like:

E0320 17:57:00.150739       1 reflector.go:138] pkg/kates/client.go:466: Failed to watch *unstructured.Unstructured: can't watch secrets.v1.: forbidden

with no indication of what secrets its specifically looking for