Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Vault agent fails to read KV-v2 secrets with deletion_time #9898

Closed
leongyh opened this issue Sep 4, 2020 · 4 comments · Fixed by #25387
Closed

Vault agent fails to read KV-v2 secrets with deletion_time #9898

leongyh opened this issue Sep 4, 2020 · 4 comments · Fixed by #25387

Comments

@leongyh
Copy link

leongyh commented Sep 4, 2020

Describe the bug
KV-v2 secrets with a "deletion_time" flag cannot be discovered and injected.

To Reproduce
Steps to reproduce the behavior:

  1. Create a KV-v2 secret engine with delete_version_after set to a non default, such as '1hr'.
  2. Create a secret in the mount.
  3. Deploy your service with annotations referencing the newly created secret. The agent will fail to find and load the secret. Describing the secret:
> vault read kv-v2/test/secret2
Key      Value                                                                                                                           
data     {"hi":"bye"}                                                                                                                    
metadata {"created_time":"2020-09-02T22:07:35.707803228Z","deletion_time":"2020-09-02T23:07:35.707803228Z","destroyed":false,"version":1}

Here is the agent-injector-init logs:

2020/09/02 22:08:30.090179 [WARN] (view) vault.read(kv-v2/test/secret2): no secret exists at kv-v2/test/data/secret2 (retry attempt 8 after "32s"

Application deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: "my-microservice"
 labels:
   app: nginx
spec:
 replicas: 1
 selector:
   matchLabels:
     app: nginx
 template:
   metadata:
     annotations:
       vault.hashicorp.com/agent-inject: "true"
       vault.hashicorp.com/role: "kubernetes-auth-vault-role"
       vault.hashicorp.com/agent-inject-secret-blah.txt: kv-v2/test/secret2
     labels:
       app: nginx
   spec:
     serviceAccountName: sample-service
     containers:
     - name: nginx
       image: nginx:1.14.2
       ports:
       - containerPort: 80

Expected behavior
The happy path where the secret did not have a deletion_time looks like this:

> vault read kv-v2/test/secret1
Key      Value                                                                                             
data     {"hey":"bae"}                                                                                     
metadata {"created_time":"2020-09-02T21:50:52.560873843Z","deletion_time":"","destroyed":false,"version":1}

The pod has the injected secret and all is well.

Environment

  • Kubernetes version: 1.15
    • Distribution or cloud vendor (OpenShift, EKS, GKE, AKS, etc.): EKS
    • Other configuration options or runtime services (istio, etc.): Nothing worth mentioning
  • vault-k8s version: 0.5.0

Additional context
We installed the injector via the vault-helm helm chart. Vault is an external instance.

@eikenb
Copy link
Contributor

eikenb commented Sep 11, 2020

I think the primary problem here is the re-use/overloading of the deletion_time field. It is used both as a deletion flag and as a scheduled event. This means downstream clock-skew will break the use of these secrets as any skew will mean you either are using invalid/deleted keys or you start ignoring un-deleted keys. As I'm having to pick one in consul-template to address this I'm going to mention clock skew in the docs and probably use some time padding to fix small skews (ie. I'm going to try to keep the skew impact to the latter, ignoring, case).

@eikenb
Copy link
Contributor

eikenb commented Sep 11, 2020

There is more discussion on this (and how it relates to consul-template) in hashicorp/vault-k8s#176

@VioletHynes
Copy link
Contributor

Hey! Thanks for bearing with us. The issue has been identified and the bug has been fixed, and should be in the next release.

@VioletHynes
Copy link
Contributor

VioletHynes commented Feb 14, 2024

Looks like I missed the boat slightly for 1.16.0, but this fix should be in 1.15.6 and 1.16.1.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants