Vault agent fails to read KV-v2 secrets with deletion_time

[leongyh]

Describe the bug
KV-v2 secrets with a "deletion_time" flag cannot be discovered and injected.

To Reproduce
Steps to reproduce the behavior:

1. Create a KV-v2 secret engine with delete_version_after set to a non default, such as '1hr'.
2. Create a secret in the mount.
3. Deploy your service with annotations referencing the newly created secret. The agent will fail to find and load the secret. Describing the secret:

> vault read kv-v2/test/secret2
Key      Value                                                                                                                           
data     {"hi":"bye"}                                                                                                                    
metadata {"created_time":"2020-09-02T22:07:35.707803228Z","deletion_time":"2020-09-02T23:07:35.707803228Z","destroyed":false,"version":1}

Here is the agent-injector-init logs:

2020/09/02 22:08:30.090179 [WARN] (view) vault.read(kv-v2/test/secret2): no secret exists at kv-v2/test/data/secret2 (retry attempt 8 after "32s"

Application deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: "my-microservice"
 labels:
   app: nginx
spec:
 replicas: 1
 selector:
   matchLabels:
     app: nginx
 template:
   metadata:
     annotations:
       vault.hashicorp.com/agent-inject: "true"
       vault.hashicorp.com/role: "kubernetes-auth-vault-role"
       vault.hashicorp.com/agent-inject-secret-blah.txt: kv-v2/test/secret2
     labels:
       app: nginx
   spec:
     serviceAccountName: sample-service
     containers:
     - name: nginx
       image: nginx:1.14.2
       ports:
       - containerPort: 80

Expected behavior
The happy path where the secret did not have a deletion_time looks like this:

> vault read kv-v2/test/secret1
Key      Value                                                                                             
data     {"hey":"bae"}                                                                                     
metadata {"created_time":"2020-09-02T21:50:52.560873843Z","deletion_time":"","destroyed":false,"version":1}

The pod has the injected secret and all is well.

Environment

• Kubernetes version: 1.15
  • Distribution or cloud vendor (OpenShift, EKS, GKE, AKS, etc.): EKS
  • Other configuration options or runtime services (istio, etc.): Nothing worth mentioning
• vault-k8s version: 0.5.0

Additional context
We installed the injector via the vault-helm helm chart. Vault is an external instance.

[eikenb]

I think the primary problem here is the re-use/overloading of the deletion_time field. It is used both as a deletion flag and as a scheduled event. This means downstream clock-skew will break the use of these secrets as any skew will mean you either are using invalid/deleted keys or you start ignoring un-deleted keys. As I'm having to pick one in consul-template to address this I'm going to mention clock skew in the docs and probably use some time padding to fix small skews (ie. I'm going to try to keep the skew impact to the latter, ignoring, case).

[eikenb]

There is more discussion on this (and how it relates to consul-template) in hashicorp/vault-k8s#176

[VioletHynes]

Hey! Thanks for bearing with us. The issue has been identified and the bug has been fixed, and should be in the next release.

[VioletHynes]

Looks like I missed the boat slightly for 1.16.0, but this fix should be in 1.15.6 and 1.16.1.