Auto-refresh MinIO credentials in notebooks (/vault)

[brendangadd]

Ensure that notebook servers always have valid MinIO credentials from vault. This issue applies to credentials injected under /vault.

Epic: #194

[zachomedia]

The issue with running a command on /vault is that while there is an annotation to run a command when the credentials are updated, it runs within the Vault sidecar and not on the user pod. I'm not sure what the fix is for this.

[blairdrummond]

This kinda seems like a feature Vault should make... But in the meantime, I guess it's a hack, but what if we just used something like

https://linux.die.net/man/5/incrontab

Edit: If we were feeling crazy, I wonder if you could even mount the incrontab file as a volume from a sidecar

[brendangadd]

I didn't realize, but the vault sidecar is successfully updating the credentials, including updating the associated files in the notebook server (thanks for providing these details @zachomedia). So CLI users just need to re-source the file, and people writing code should transparently use the updated creds.

There are edge cases here where token refresh can cause issues, but I think we're at a reasonable place in our current state.

If we do decide to tackle this further, then I think we would want to do so with documentation. Particularly now that we auto-mount MinIO storage into the file system.

/cc @blairdrummond