-
Notifications
You must be signed in to change notification settings - Fork 4.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
KV Engine: Recursively list keys #5275
Comments
This is a very useful feature, especially for the HTTP API |
Adds `-recursive', `-depth', `-filter' and `-concurrent' flags to the vault CLI's `kv list' subcommand. Description of the flags: * recursive Recursively list data for a given path. * depth Specifies the depth for recursive listing. * filter Specifies a regular expression for filtering paths. * concurrent Specifies the number of concurrent recursions to run. Reference: hashicorp#5275.
If anyone stumble upon this I made a little script (not really efficient) while we wait for a native call. Not battle tested but good enough!
#!/usr/bin/env bash
# Recursive function that will
# - List all the secrets in the given $path
# - Call itself for all path values in the given $path
function traverse {
local -r path="$1"
result=$(vault kv list -format=json $path 2>&1)
status=$?
if [ ! $status -eq 0 ];
then
if [[ $result =~ "permission denied" ]]; then
return
fi
>&2 echo "$result"
fi
for secret in $(echo "$result" | jq -r '.[]'); do
if [[ "$secret" == */ ]]; then
traverse "$path$secret"
else
echo "$path$secret"
fi
done
}
# Iterate on all kv engines or start from the path provided by the user
if [[ "$1" ]]; then
# Make sure the path always end with '/'
vaults=("${1%"/"}/")
else
vaults=$(vault secrets list -format=json | jq -r 'to_entries[] | select(.value.type =="kv") | .key')
fi
for vault in $vaults; do
traverse $vault
done |
Also interested in this kind of feature. |
For anyone ending up here, I created a small cli to perform recursive kv read/list operations while we wait for the native solution. |
@agaudreault-jive, thanks! |
Issues that are not reproducible and/or have not had any interaction for a long time are stale issues. Sometimes even the valid issues remain stale lacking traction either by the maintainers or the community. In order to provide faster responses and better engagement with the community, we strive to keep the issue tracker clean and the issue count low. In this regard, our current policy is to close stale issues after 30 days. If a feature request is being closed, it means that it is not on the product roadmap. Closed issues will still be indexed and available for future viewers. If users feel that the issue is still relevant but is wrongly closed, we encourage reopening them. Please refer to our contributing guidelines for details on issue lifecycle. |
Would love this feature |
I would love to have this natively, this is how I implemented it in Go for a project, it's probably not optimal or anything, but it worked for me. var secretListPath []string
func isNil(v interface{}) bool {
return v == nil || (reflect.ValueOf(v).Kind() == reflect.Ptr && reflect.ValueOf(v).IsNil())
}
// ListSecret returns a list of secrets from Vault
func ListSecret(vaultCli *vault.Client, path string) (*vault.Secret, error) {
secret, err := vaultCli.Logical().List(path)
if err != nil {
log.Println("Couldn't list from the Vault.")
}
if isNil(secret) {
log.Printf("Couldn't list %s from the Vault.", path)
}
return secret, err
}
// RecursiveListSecret returns a list of secrets paths from Vault
func RecursiveListSecret(vaultCli *vault.Client, path string) []string {
secretList, err := ListSecret(vaultCli, path)
if err == nil && secretList != nil {
for _, secret := range secretList.Data["keys"].([]interface{}) {
if strings.HasSuffix(secret.(string), "/") {
RecursiveListSecret(vaultCli, path+secret.(string))
} else {
secretListPath = append([]string{strings.Replace(path, "metadata", "data", -1) + secret.(string)}, secretListPath...)
}
}
}
return secretListPath
} |
@rnsc Just in case you didn't notice my comment above and it can help, I wrote a small go based cli for this (#5275 (comment)) |
Adding my vote for this feature please. |
Ditto! |
One more vote for search |
please implement this. It has been 4 years! |
This is a must have, listing all secrets now is a major issue. |
if 103 people upvoted this feature request, maybe it's about time to start adding this feature 😸 |
if have not install jq command then |
This script doesn't work if you have spaces in the path name. Even with passing the argument with double quotes. |
Hi folks. We are planning to add this feature soon, but have some questions for the people on this thread about expected behavior depending on LIST and READ permissions on secrets and paths:
|
Good news! For your question, I would say:
|
|
For me it should silently not return anything that it cannot list or read in the sub tree. If you want to recurse from a top level path and you only have read access to a secret from two levels down I'd argue returning a 201 would be the correct behaviour. As it wouldn't show up in the UI either would it? That's just how I would implement this for me though with the predicate that people know that recursively iterating over a path that you don't have the appropriate ACL for your token set to do is a user error and that given how this works we cannot hold the hands of the user too much without implementing insane logic. |
I think I disagree with some of the suggestions above, mainly because the topic of this issue is about recursively
|
@finnstech This is great news. Is there any low/high confidence guess about when such functionality woudl be released? This quarter, this year? |
Is there any progress on this issue? I'm keen on using this feature. |
Also interested in this feature |
Same here. |
very interested in this feature |
Any updates on this? This would be a highly beneficial feature for our bank-vaults project. |
How can you not have search yet? |
@oyvhvi exactly! At the moment we are implementing our own vault-cli with all workarounds due to this missing feature. |
Just adding another vote for this. I know search in Vault has been a highly contentious topic for literally years at this point, but even some basic search functionality in the KV engine would be incredibly beneficial. As a mitigation for all the performance concerns that are always brought up, how about making it an option one could enable or disable on a given KV engine? |
I built on your code, I had a use case to export all the KV secrets for a given path and save it locally in a #!/usr/bin/env bash
# vault-kv-export.sh
set -eo pipefail
readonly ARB_TEMP_SECRETS_FILE="arbitrary_temp_secrets.json"
readonly TEMP_SECRETS_FILE="temp_secrets.json"
readonly SECRETS_FILE="secrets.json"
log() {
local log_type="$1"
local message="$2"
local timestamp
timestamp=$(date +"%Y-%m-%d %H:%M:%S")
echo "[$log_type] [$timestamp] $message"
}
log_info() {
log "INFO" "$1"
}
log_error() {
log "ERROR" "$1"
exit 1
}
traverse() {
local path="$1"
local result
local headers=()
if [[ -n "${CF_TOKEN}" ]]; then
headers+=("-header" "cf-access-token=${CF_TOKEN}")
fi
result=$(vault kv list -format=json "${headers[@]}" "${path}" 2>&1) || log_error "Failed to list secrets: ${result}"
while IFS= read -r secret; do
if [[ "${secret}" == */ ]]; then
traverse "${path}${secret}"
else
local secret_data
secret_data=$(vault kv get -format=json "${headers[@]}" "${path}${secret}" | jq -r '.data') || log_error "Failed to get secret data: ${secret_data}"
if [[ "${secret_data}" != "null" ]]; then
echo "{\"path\":\"${path}${secret}\",\"value\":{\"data\":${secret_data}}}," >>"${ARB_TEMP_SECRETS_FILE}"
fi
fi
done < <(echo "${result}" | jq -r '.[]')
}
main() {
log_info "Starting secrets retrieval process."
[[ -f "${ARB_TEMP_SECRETS_FILE}" ]] && rm -f "${ARB_TEMP_SECRETS_FILE}"
[[ -f "${TEMP_SECRETS_FILE}" ]] && rm -f "${TEMP_SECRETS_FILE}"
[[ -f "${SECRETS_FILE}" ]] && rm -f "${SECRETS_FILE}"
if [[ -n "${CF_TOKEN}" ]]; then
log_info "CF_TOKEN detected."
else
log_info "CF_TOKEN not provided. Headers will not be attached to Vault requests."
fi
local vaults
if [[ "$1" ]]; then
vaults=("${1%"/"}/")
log_info "Retrieving all secrets under ${vaults[*]}.."
else
local headers=()
if [[ -n "${CF_TOKEN}" ]]; then
headers+=("-header" "cf-access-token=${CF_TOKEN}")
fi
log_info "No secret engine provided. Retrieving all secrets.."
result=$(vault secrets list -format=json "${headers[@]}" 2>&1) || log_error "Failed to list secrets engines: ${result}"
mapfile -t vaults < <(echo "${result}" | jq -r 'to_entries[] | select(.value.type=="kv") | .key')
fi
for vault in "${vaults[@]}"; do
traverse "${vault}"
done
echo "[" >"${TEMP_SECRETS_FILE}"
sed '$s/,$//' "${ARB_TEMP_SECRETS_FILE}" >>"${TEMP_SECRETS_FILE}"
echo "]" >>"${TEMP_SECRETS_FILE}"
jq . "${TEMP_SECRETS_FILE}" >"${SECRETS_FILE}"
rm "${ARB_TEMP_SECRETS_FILE}" "${TEMP_SECRETS_FILE}"
log_info "Secrets retrieval completed and saved to ${SECRETS_FILE}"
}
[[ "$0" == "${BASH_SOURCE[0]}" ]] && main "$@"
a sample output: $ ./vault-kv-export.sh secret2/confluent_cloud
[INFO] [2023-11-03 15:06:15] Starting secrets retrieval process.
[INFO] [2023-11-03 15:06:15] CF_TOKEN detected.
[INFO] [2023-11-03 15:06:15] Retrieving all secrets under secret2/confluent_cloud/..
[INFO] [2023-11-03 15:06:27] Secrets retrieval completed and saved to secrets.json reading $ cat secrets.json
[
{
"path": "secret2/confluent_cloud/global-creds",
"value": {
"data": {
"data": {
"confluent_cloud_token": "REDACTED"
},
"metadata": {
"created_time": "2023-11-02T13:14:09.616943342Z",
"custom_metadata": null,
"deletion_time": "",
"destroyed": false,
"version": 1
}
}
}
},
{
"path": "secret2/confluent_cloud/service/credentials",
"value": {
"data": {
"data": {
"api_key": "REDACTED",
"api_secret": "REDACTED"
},
"metadata": {
"created_time": "2023-11-02T13:14:10.336792836Z",
"custom_metadata": null,
"deletion_time": "",
"destroyed": false,
"version": 1
}
}
}
},
... Hope it helps |
I am adding my vote, is there any news about when we can have it ? or at least what are the best alternatives for the moment? |
would also like to see this implemented |
It would be great to get an update on this. The last official comment on this was March. Secret cycling without this is unbelievably frustrating, especially for complex systems you weren't around for at the beginning (e.g. is this value still used somewhere?). |
Adding my vote - this would be super helpful for our use case! |
Hi everyone, I managed to implement a server-side recursive secret search feature by adding a new API endpoint to the KV engine. You can check my work here if you're interested for carrying further testing: https://github.com/kosmos-education/vault-plugin-secrets-kv As I mentioned in the repository, please note that this is a highly experimental feature which could potentially lead to stability issues or vulnerabilities. |
Any progress or developments regarding this? This feature would provide significant benefits. |
I know this feature request is focused on CLI/API, but are there any plans to provide a tree ui for KV mounts on top of this upcoming work? I think that would make the UI much more usable for ops that need to deal on a daily basis with a lot of static secrets. Also interested in knowing what the community thinks about this. Cheers. |
Using vault-kv-search and fzf creates a nice way to browse through kv secrets
|
@adrianlzt What's the |
faq is like jq but with form input/output formats (yaml between them). I'm finally using (as the first get of all keys in my environment is quite slow):
|
6 years and no basic implementation of this feature? |
Hello folks! We appreciate the feedback here. Our product managers are looking for folks willing to chat in deeper detail (ie via Zoom). Please email me (heather dot ezell, at hashicorp dot com) if you're willing to set up a deeper-dive conversation around your needs. |
(Corrected my email address.) |
Is your feature request related to a problem? Please describe.
With a KV engine, if I want to list all keys in the directory
/foo/
, it only returns keys directly under/foo/
For example, if I have the following keys:
A LIST operation on
/foo/
returnssome_key
andbar/
, while I would like to havesome_key
andbar/some_other_key
Describe the solution you'd like
Add a parameter to either recursively return ALL keys in the provided path.
Describe alternatives you've considered
Another way to do it would be to add a parameter specifying the depth up to which look recursively for keys.
The text was updated successfully, but these errors were encountered: