Add flags to recursively list secrets for KV Engine.

[clickyotomy]

Adds `-recursive', `-depth', `-filter' and `-concurrent' flags to
vault CLI's `kv list' subcommand.

Description of the flags:
    * recursive     Recursively list data for a given path.
    * depth         Specifies the depth for recursive listing.
    * filter        Specifies a regular expression for filtering paths.
    * concurrent    Specifies the number of concurrent recursions to run.

Reference: #5275.

[clickyotomy]

👋 - Not sure why TEST_COMMAND='travis_wait 75 make testracetravis' fails. Ran make testrace locally and it passed.

[clickyotomy]

@jefferai - Can you please review?

[jefferai]

We need to discuss internally whether or not we want this functionality and if so whether it should be client or server side.

[v6]

Thanks for considering it guys

[jcrowthe]

@jefferai Regarding your last comment about serverside or clientside.

This feature would be ideal for other projects that integrate with Vault. Currently in order to get a picture of what keys exist you must recursively traverse the entire tree LISTing on each "folder". While it could be done clientside, it would also be costly for larger vault installations. While doing this serverside would still incur a cost, it is in-memory which would be an orders-of-magnitude smaller cost considering the TCP/TLS connections that would need to be established per path.

In the end, both client- and server-side have use cases, but client-side has the ability to be done today through the API. Given there are several tickets regarding this feature, it appears an official server-side implementation would be a welcome addition to the project. Thoughts?

[clickyotomy]

Hello! Any update on this?

[catsby]

A similar PR was opened in the KV repo, but subsequently closed:

• Recursive list feature vault-plugin-secrets-kv#38

The reasoning for closing involves possible acl exploitation. Do you feel like that's a possible issue with this implementation?

[michelvocks]

hashicorp/vault-plugin-secrets-kv#38 ACL exploit was possible because the implementation happened on the server-side. This PR implements the functionality via the client-side where an ACL exploit is not possible.

I personally think that both hashicorp/vault-plugin-secrets-kv#38 and this PR are not the right way to implement this feature due to the fact that current LIST operations for our storage backends are not optimized to do a recursive list. I agree with @jcrowthe that this feature is much needed but due to complexity reasons, this should be taken over by an HashiCorp employee. @jefferai @briankassouf It would be great to get your input here.

[mcouthon]

Is there a semi-canonical way to do this somewhere, until an official solution is implemented? Maybe some reference implementation?

[krilor]

Is there a semi-canonical way to do this somewhere, until an official solution is implemented? Maybe some reference implementation?

I'm not sure that it is "semi-canonical", but I'm using this little bash script:

#!/bin/bash
# Usage: vault-rlist.sh secret/some/path/
# Trailing slash is mandatory!

walk() {
    wd=$1

    if [ "${wd: -1}" != "/" ]; then
        echo $wd
        exit
    fi

    vault kv list -format=json $wd | jq -r '.[]' | xargs -r -I % bash -c "walk $wd%"
}
export -f walk
walk $1

If I want to hide errors, I do vault-rlist.sh secret/ 2>/dev/null

[heatherezell]

Hi folks! I'm going to go ahead and close this PR - please feel free to re-open it (and rebase off of main) if the questions can be answered and you'd like to continue working on it. Thanks!