User can't recover signed identities on Helm since v13

[hugoShaka]

What would you like Teleport to do?

As a user, I want to be able to sign an identity for a server or a Teleport user by exec-ing in the auth pods. This was working until v13. I want tctl to support sending the signed content to stdout.

I also want the workarounds and proper way to sign identities to be documented (see workarounds).

What problem does this solve?

Trust bootstrapping. A lot of users used kubectl exec -it deployment/auth -- tctl auth sign ... to sign various things like the teleport plugin identities or db/host certs.

Until v13, we were using debian-based images, with tar in them. Now that we are using distroless-based images, we don't have tar anymore and cannot invoke kubectl cp by default.

If a workaround exists, please include it.

• use a different image flavour (distroless-debug contains a shell and tar)
• login using your local tctl (tsh login, then run locally tctl auth sign). This provides betters security and auditability guarantees as the actions are logged as your user and RBAC is enforced. This approach should be preferred when possible.
• hack to send tctl's output to stdout (use -o /proc/self/fd/1 --overwrite). This doesn't work when the output of tctl auth sign is multiple files (e.g. DB certs)

[tcsc]

Is something like this what you're after?

❯ kubectl --namespace teleport exec deploy/teleport-auth -- tctl auth sign --user admin-guy --format openssh -o ssh-adminguy
The credentials have been written to ssh-adminguy, ssh-adminguy-cert.pub
❯ kubectl --namespace teleport exec deploy/teleport-auth -- tar c ssh-adminguy ssh-adminguy-cert.pub > adminguy-local.tar
❯ tar -xvf adminguy-local.tar
x ssh-adminguy
x ssh-adminguy-cert.pub

[tigrato]

@tcsc

The goal is to be able to use kubectl cp to fetch the resources from the pod to the local machine.
Internally, kubectl cp uses tar to compress and transfer the files.

❯ kubectl --namespace teleport exec deploy/teleport-auth -- tctl auth sign --user admin-guy --format openssh -o ssh-adminguy
The credentials have been written to ssh-adminguy, ssh-adminguy-cert.pub
❯ kubectl --namespace teleport cp deploy/teleport-auth:./ssh-adminguy ./localDir

[tcsc]

OK, so I've built and tested a distroless image with tar and it's dependencies, and both of the above approaches work:

Given some files we want to extract from a pod, for the sake of an example created via

❯ kubectl -n teleport exec deploy/teleport-auth -- tctl auth sign --user admin-guy --format openssh -o adminguy
The credentials have been written to adminguy, adminguy-cert.pub

kubectl cp

❯ kubectl cp -n teleport teleport-auth-67c6844544-8jkrp:adminguy adminguy.local
❯ ls -la adminguy.local
-rw-r--r--  1 trent  staff  1679 24 Jul 11:59 adminguy.local

Write tar to stdout:

❯ mkdir local
❯ kubectl -n teleport exec deploy/teleport-auth -- tar cv adminguy adminguy-cert.pub | tar xv -C local
adminguy
adminguy-cert.pub
x adminguy
x adminguy-cert.pub
❯ ls -l local
total 16
-rw-------  1 trent  staff  1679 24 Jul 11:39 adminguy
-rw-------  1 trent  staff  2150 24 Jul 11:39 adminguy-cert.pub

Oops
So while we can get stuff out of the pod easily, this still leaves the issue of cleaning up after copying them out of the pod because the image has no rm. Given the way that the image is built, the obvious way to add rm implies adding the whole coreutils package, and at this point we may as well just use the debug image with busybox installed.

Question re: workarounds:
Are there scenarios where workaround #2 is inappropriate? Putting aside the fact that this is a breaking change and may break customer workflows; if workaround #2 is the preferred method in the first place, should we be encouraging that (and managing the breaking change) rather than making it easier to slip around access auditing and suchlike?

[tcsc]

If we do end up wanting tctl to write files to stdout, we probably want it to spit out a tar archive in cases where it generates multiple files, yes? It's pretty easy to imagine something like:

❯ kubectl  exec -n teleport deploy/auth -- tctl auth sign --user admin-guy --format openssh -o - | tar xv
x adminguy
x adminguy-cert.pub

Not sure whether handling single-file exports with tar (for consistency) or as their natural format (for immediate use) is the better user experience.

Edit: see #29451 for experimental implementation