Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Secrets Import documentation #25594

Merged
merged 24 commits into from
Mar 27, 2024
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
b0724f2
Start import docs
robmonte Feb 22, 2024
d793357
Use hideClipboard block on output
robmonte Feb 22, 2024
eaf2a9e
Fix missed line
robmonte Feb 22, 2024
51c9268
Reorganize mappings and source docs, fill out more of the info
robmonte Feb 26, 2024
ce6f30c
Remove copied section
robmonte Feb 28, 2024
22c9a1e
Apply suggestions from code review
robmonte Mar 25, 2024
194a621
Apply suggestions from code review
robmonte Mar 25, 2024
da4323c
Apply suggestions from code review
robmonte Mar 25, 2024
34a5f71
Apply suggestions from code review
robmonte Mar 25, 2024
302831c
Apply suggestions from code review
robmonte Mar 25, 2024
2435ae0
Fix typo
robmonte Mar 25, 2024
9aa1b51
Update website/content/docs/import/index.mdx
robmonte Mar 25, 2024
7af65e8
Apply suggestions from code review
robmonte Mar 25, 2024
11c8ae8
Add experimental partial, apply more suggestions that can't be autoap…
robmonte Mar 25, 2024
f8232dc
Align remaining HCL blocks
robmonte Mar 25, 2024
9735bc7
Fix typos, wording, add links, fill in small bits of info
robmonte Mar 25, 2024
38fb9d7
Update wording and formatting for gcp source and mappings
robmonte Mar 25, 2024
a94fd7c
Merge branch 'main' into docs/secrets-import
robmonte Mar 26, 2024
4e6f242
Change experimental to alpha
robmonte Mar 26, 2024
d14f0e1
Change list tag to alpha
robmonte Mar 26, 2024
2fa7329
Fix file
robmonte Mar 26, 2024
908c858
Apply suggestions from code review
robmonte Mar 26, 2024
5359662
Remove extra line, trigger CI rerun
robmonte Mar 27, 2024
1285247
Merge branch 'main' into docs/secrets-import
robmonte Mar 27, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
87 changes: 87 additions & 0 deletions website/content/docs/commands/operator/import.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
---
layout: docs
page_title: operator import - Command
description: >-
The "operator import" command imports secrets from external systems
in to Vault.
---

# operator import

The `operator import` command imports secrets from external systems in to Vault.
Secrets with the same name at the same storage path will be overwritten upon import.

~> **NOTE:** This has the potential to be a long-running process. Import plans can be
written to read from as many sources as desired and each source may migrate as many
secrets as requested based on filters.
robmonte marked this conversation as resolved.
Show resolved Hide resolved
robmonte marked this conversation as resolved.
Show resolved Hide resolved

## Examples

Execute an import config file named import.hcl to generate an import plan:
robmonte marked this conversation as resolved.
Show resolved Hide resolved
robmonte marked this conversation as resolved.
Show resolved Hide resolved

```shell-session
$ vault operator import -config import.hcl plan
```

Output:

<CodeBlockConfig hideClipboard>

-----------
Import plan
-----------
The following namespaces are missing:
* ns-1/

The following mounts are missing:
* ns-1/mount-1

Secrets to be imported from the source "my-dest-1":
* secret-1
* secret-2

</CodeBlockConfig>

## Configuration

The `operator import` command uses a dedicated configuration file to specify the source,
destination, and mapping rules.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we explain what are these and what are the types & fields available in the HCL schema?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like instead of explaining here, it should just link to the main docs pages explaining the feature. What do you think?


```hcl
source_gcp {
name = "my-gcp-source-1"
credentials = "@/path/to/service-account-key.json"
}

destination_vault {
name = "my-dest-1"
address = "http://127.0.0.1:8200/"
token = "root"
namespace = "ns-1"
mount = "mount-1"
}

mapping_passthrough {
name = "my-map-1"
source = "my-gcp-1"
destination = "my-dest-1"
priority = 1
}
```
robmonte marked this conversation as resolved.
Show resolved Hide resolved

## Usage
robmonte marked this conversation as resolved.
Show resolved Hide resolved

The following flags are available for the `operator import` command.
robmonte marked this conversation as resolved.
Show resolved Hide resolved

- `-config` `(string: "<import.hcl>")` - Path to the import configuration HCL file. If unspecified,
a file named import.hcl will be automatically looked for.
robmonte marked this conversation as resolved.
Show resolved Hide resolved
robmonte marked this conversation as resolved.
Show resolved Hide resolved

- `auto-approve` `(bool: <false>)` - Automatically skips the user-input requirement of "yes" when
running the "apply" command.
robmonte marked this conversation as resolved.
Show resolved Hide resolved

- `auto-create` `(bool: <false>)` - Automatically creates any missing namespaces and mounts when
"running the "apply" command.
robmonte marked this conversation as resolved.
Show resolved Hide resolved

- `-log-level` ((#\_log_level)) `(string: "info")` - Log verbosity level. Supported values (in
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does the # mean?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't actually know, I copied this flag description from another command doc that has it too. There's a handful of files that have this, for example website/content/docs/commands/server.mdx

order of descending detail) are `trace`, `debug`, `info`, `warn`, and `error`. This can
also be specified via the `VAULT_LOG_LEVEL` environment variable.
robmonte marked this conversation as resolved.
Show resolved Hide resolved
1 change: 1 addition & 0 deletions website/content/docs/commands/operator/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ Usage: vault operator <subcommand> [options] [args]

Subcommands:
generate-root Generates a new root token
import Import secrets from external systems into Vault
init Initializes a server
key-status Provides information about the active encryption key
rekey Generates new unseal keys
Expand Down
41 changes: 41 additions & 0 deletions website/content/docs/import/gcpsm.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
---
layout: docs
page_title: Google Cloud Platform Secret Manager - Secrets Import Source
robmonte marked this conversation as resolved.
Show resolved Hide resolved
description: The Google Cloud Platform Secret Manager source imports secrets from GCP to Vault.
robmonte marked this conversation as resolved.
Show resolved Hide resolved
---
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The general documentation typically uses a walkthrough style that guides the user through a full-working example from start to finish. Do you think the same pattern would be useful for Secrets Import?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had first copied our usual format based on other docs to base these on, but I started to notice when writing the GCP page that other source pages would be nearly identical. The only section that would be different is explaining the source block with the rest being unchanged. I figure it makes sense to have a small page for each source block, a page for the mappings, and put the destination info on the index page since we are only dealing with Vault as a destination.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Walking folks through a full working example is specific to tutorials. Docs have a more focused purpose. Overviews provide the who/what/where/when/why/how of a feature or function. And how-to guides provide explicit-but-generalized steps for solving specific problems.

If this doc is an overview. Focus on explaining:

  • who can, or would want to, use the feature/functionality
  • what problem the feature/functionality solves
  • where the feature/function is configured or managed
  • when the feature/function is and isn't useful
  • why the feature/function is better than the alternative
  • how it solves the problem at a high level (if a detailed discussion of the the nuts-and-bolts is needed, consider a full technical overview as a separate document)


# GCP import source

The GCP source enables users to import secret data from GCP Secret Manager into their Vault instance. To be able
to import secret data from here into your Vault destination, the following permissions are required:
robmonte marked this conversation as resolved.
Show resolved Hide resolved

```shell-session
"secretmanager.secrets.list",
"secretmanager.versions.access",
```
robmonte marked this conversation as resolved.
Show resolved Hide resolved

## Argument reference
robmonte marked this conversation as resolved.
Show resolved Hide resolved

* `name` - (Required) The name of the source block used to internally reference this source inside of a mapping.
robmonte marked this conversation as resolved.
Show resolved Hide resolved
robmonte marked this conversation as resolved.
Show resolved Hide resolved

* `credentials` - (Optional) The path to the service account key credentials file for the service account with the ncessary
robmonte marked this conversation as resolved.
Show resolved Hide resolved
permissions. If `credentials` is set, then `vault_mount_path` and `vault_role_name` must be unset.

* `vault_mount_path` - (Optional) The Vault mount path to a preconfigured GCP secrets engine that is used to generate a
dynamic credential for the importer to use. If `vault_mount_path` or `vault_role_name` are set, then `credentials` must be unset.
robmonte marked this conversation as resolved.
Show resolved Hide resolved

* `vault_role_name` - (Optional) The Vault role's name that exists in the preconfigured GCP secrets engine mount that is
used to generate a dynamic credential for the importer to use. If `vault_role_name` or `vault_mount_path` are set, then
`credentials` must be unset.
robmonte marked this conversation as resolved.
Show resolved Hide resolved

## Example

In this example, a GCP source is defined and configured to use
robmonte marked this conversation as resolved.
Show resolved Hide resolved

```hcl
source_gcp {
name = "my-gcp-source-1"
secrets_engine_mount = "gcp"
secrets_engine_role = "my-gcp-role-1"
robmonte marked this conversation as resolved.
Show resolved Hide resolved
}
```
122 changes: 122 additions & 0 deletions website/content/docs/import/index.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
---
layout: docs
page_title: Secrets import
description: Secrets import allows you to safely onboard secrets from external sources into Vault KV for management.
---


# Secrets import

<EnterpriseAlert product="vault" />
robmonte marked this conversation as resolved.
Show resolved Hide resolved

@include 'alerts/beta.mdx'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment here, we might need guidance to create an alpha or experimental alert.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we've publicly documented alpha or experimental features before. If that's needed here, feel free to create a new partial by using the beta partial as a template.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added an experimental that I'll push up soon to be reviewed.


Having collections of secrets across various external systems can present several challenges to an organization, such as
increased operational overhead, a greater risk of secrets sprawl, and more. Vault can offer a single source of truth (SSOT)
solution to managing these secrets, providing increased security and reducing that overhead, yet the effort required
to migrate all your secrets from multiple external sources into Vault can be steep. To aid in this onboarding process,
secrets import provides a mechanism to automate it.
robmonte marked this conversation as resolved.
Show resolved Hide resolved

A secret that is imported from an external system is stored in Vault's KVv2 Secrets Engine. The import process is defined
an import plan, which is an HCL file. Using HCL enables users to declaratively configure which of their secrets will be read
from an external source and where exactly they will be stored in Vault.
robmonte marked this conversation as resolved.
Show resolved Hide resolved

There are three main types of HCL blocks used to configure a secrets import plan. They are source blocks, the Vault destination
block, and mapping blocks.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we loop in product or marketing here to validate the wording fits with their vision? It feels like a snippet they'd normally provide or would like to review.

robmonte marked this conversation as resolved.
Show resolved Hide resolved

## Vault destinations
robmonte marked this conversation as resolved.
Show resolved Hide resolved
robmonte marked this conversation as resolved.
Show resolved Hide resolved

Secrets that get imported are stored into a Vault KVv2 Secrets Engine mount. The destination will accept the mount path of
the desired KVv2 mount and optionally a namespace. This specifies the exact location to store the secrets in. A destination
is defined by an HCL block in the format of `destination_vault`.
robmonte marked this conversation as resolved.
Show resolved Hide resolved

### HCL syntax

#### Argument reference
robmonte marked this conversation as resolved.
Show resolved Hide resolved

The following arguments are supported:
robmonte marked this conversation as resolved.
Show resolved Hide resolved

* `name` - (Required) The name of the Vault destination block used to internally reference this destination inside of a mapping.
robmonte marked this conversation as resolved.
Show resolved Hide resolved
robmonte marked this conversation as resolved.
Show resolved Hide resolved

* `address` - (Optional) The address of the Vault server with the KVv2 Secrets Engine for which the secrets will be imported into.
robmonte marked this conversation as resolved.
Show resolved Hide resolved

* `token` - (Optional) A token for authentication to the Vault server located at the specified address.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we document the address and token if they are always overridden? There is no reason for a Vault user to define these in their config file.

Copy link
Member Author

@robmonte robmonte Mar 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't believe they are always overridden, I think they are only loaded from the client if they are unset in the plan, hence why it's optional. Do you know if I am forgetting something?

robmonte marked this conversation as resolved.
Show resolved Hide resolved

* `namespace` - (Optional) The name of the Vault namespace containing the specified KVv2 mount.
robmonte marked this conversation as resolved.
Show resolved Hide resolved
robmonte marked this conversation as resolved.
Show resolved Hide resolved

* `mount` - (Optional) The name of the KVv2 mount in Vault for which the secrets will be imported into.
robmonte marked this conversation as resolved.
Show resolved Hide resolved
robmonte marked this conversation as resolved.
Show resolved Hide resolved

#### Example
robmonte marked this conversation as resolved.
Show resolved Hide resolved

An example `destination_vault` block may look like the following:

```hcl
destination_vault {
name = "my-dest-1"
address = "http://127.0.0.1:8200/"
token = "root"
namespace = "ns-1"
mount = "mount-1"
}
```
robmonte marked this conversation as resolved.
Show resolved Hide resolved
robmonte marked this conversation as resolved.
Show resolved Hide resolved

## Sources
robmonte marked this conversation as resolved.
Show resolved Hide resolved

Secrets can be imported from various external systems, called sources. The supported destinations are:
robmonte marked this conversation as resolved.
Show resolved Hide resolved
robmonte marked this conversation as resolved.
Show resolved Hide resolved
* [GCP Secret Manager](/vault/docs/import/gcpsm)
robmonte marked this conversation as resolved.
Show resolved Hide resolved

A source requires the credentials necessary to read the external system's secrets and their values so that it may import
them into Vault. Either credentials to the system can be directly provided, or you can leverage Vault's capabilities to
geneerate a dynamic secret for the importer to use if you have the corresponding secrets engine configured. A source is
robmonte marked this conversation as resolved.
Show resolved Hide resolved
defined by an HCL block in the format of `source_<external_system>`. For example, `source_gcp` defines a source block
for GCP Secret Manager.
robmonte marked this conversation as resolved.
Show resolved Hide resolved

### HCL syntax

Refer to a [specific source's documentation](#sources) for details on how to define all remaining arguments of a source
block for that external system type.
robmonte marked this conversation as resolved.
Show resolved Hide resolved

#### Example
robmonte marked this conversation as resolved.
Show resolved Hide resolved

A `source_<external_system>` block may look like the following, using GCP as an example:

```hcl
source_gcp {
name = "my-gcp-source-1"
credentials = "@/path/to/service-account-key.json"
}
```
robmonte marked this conversation as resolved.
Show resolved Hide resolved

## Mappings
robmonte marked this conversation as resolved.
Show resolved Hide resolved

Mappings blocks are what glue together a source block to a destination block. The supported mapping types are:
* [mapping_passthrough](/vault/docs/import/mappings#passthrough)
* [mapping_metadata](/vault/docs/import/mappings#metadata)
* [mapping_regex](/vault/docs/import/mappings#regex)

A mapping is where filtering logic is defined to control which secrets in the external system will be imported and which will
be ignored. The definition of the rule depends on the type of mapping block that is being configured. Mapping blocks are
applied in order of their given priority.

Once a secret being imported matches a single mapping's rule, it cannot match to another mapping again. A mapping is defined
by an HCL block in the format of `mapping_<filter_type>`. For example, `mapping_regex` defines a mapping block to pass
all secrets through whose secret name passes the given regular expression.
robmonte marked this conversation as resolved.
Show resolved Hide resolved

### HCL syntax

See the [mapping documentation](/vault/docs/import/mappings) to see more info regarding specific mapping types and
their arguments.

#### Example

A `mapping_<filter_type>` block may look like the following, using regex as an example:

```hcl
mapping_regex {
name = "my-map-1"
source = "my-gcp-source-1"
destination = "my-dest-1"
priority = 1
expression = "^database/.*$"
}
```
robmonte marked this conversation as resolved.
Show resolved Hide resolved
119 changes: 119 additions & 0 deletions website/content/docs/import/mappings.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
---
layout: docs
page_title: Secrets import mappings
description: Mappings lets users apply various filtering methods to secrets being imported in to Vault.
---

# Import mappings

There are multiple types of mapping blocks that can be defined in the import plan. Each type enables a
robmonte marked this conversation as resolved.
Show resolved Hide resolved
different mechanism for filtering out secrets from an external system during the execution of an import.
robmonte marked this conversation as resolved.
Show resolved Hide resolved
robmonte marked this conversation as resolved.
Show resolved Hide resolved

## Argument reference
robmonte marked this conversation as resolved.
Show resolved Hide resolved

All types of mapping blocks, no matter the type, have four shared arguments:

* `name` - (Required) The name of the mapping block used internally to track secrets imported from a source to a destination.

* `source` - (Required) The name of the source block for which this mapping's filter logic will be applied to.

* `destination` - (Required) The name of the Vault destination block for which this mapping will place the matching secrets into.

* `priority` - (Required) Controls the order in which multiple mapping blocks will be applied to the secrets being imported. The
lower the value specified, the higher the priority will be during filtering.
robmonte marked this conversation as resolved.
Show resolved Hide resolved

## Passthrough
robmonte marked this conversation as resolved.
Show resolved Hide resolved

Passthrough mapping blocks `mapping_passthrough` allow all secrets through from the specified source to the
specified destination. For example, one use case is using it as a base-case for imported secrets. By assigning
it the lowest priority in the import plan, all other mapping blocks will be applied first. Secrets that fail
to match any of the previous mappings will fall through to the passthrough block and be collected in a single
KVv2 bucket.

### Additional arguments

There are no extra arguments to specify in a `mapping_passthrough` block.

### Example

In this example, every single secret that `my-gcp-source-1` is reads from GCP Secret Manager will be imported
to the KVv2 Secrets Engine mount defined in `my-dest-1`.

```hcl
mapping_passthrough {
name = "my-map-1"
source = "my-gcp-source-1"
destination = "my-dest-1"
priority = 1
}
```

## Metadata

Metadata mapping blocks `mapping_metadata` allow secrets through from the specified source to the specified
destination if they contain matching metadata key-value pairs. Metadata is not supported in all external secret
manageement systems, and ones that do may use different terminology for metadata. For example, AWS allows tags
on secrets while [GCP](/vault/docs/import/gcpsm) allows labels.

### Additional arguments

* `tags` - (Required) A set of key-value pairs to match on secrets from the external system. All of the specified
keys must be found on a secret and all of the values must be exact matches. A key with an empty value, i.e. `""`,
must also match an empty value from the external system.
robmonte marked this conversation as resolved.
Show resolved Hide resolved

### Example

In this example, there are two metadata mappings. The first, `my-map-1`, has a priority of 1. This will only import
the secrets into the destination `my-dest-1` that contain both tag keys `database` and `importable`. Each of these
keys' values must also match to `users` and `true` respectively to be imported. The second, `my-map-2`, has a
priority of 2. Even though all the secrets in the first map would also qualify for the second map's filtering rule,
those secrets will only be imported into `my-dest-1` because of the lower priority set in the second mapping. All
remaining secrets that have the tag `importable` with a value of `true` will be imported into `my-dest-2`.

```hcl
mapping_metadata {
name = "my-map-1"
source = "my-gcp-source-1"
destination = "my-dest-1"
priority = 1
tags = {
"database" = "users"
"importable" = "true"
}
}

mapping_metadata {
name = "my-map-2"
source = "my-gcp-source-1"
destination = "my-dest-2"
priority = 2
tags = {
"importable" = "true"
}
}
```
robmonte marked this conversation as resolved.
Show resolved Hide resolved

## Regex

Regex mapping blocks `mapping_regex` allow secrets through from the specified source to the specified
destination if their secret name passes a regular expression check.

### Additional arguments

* `expression` - (Required) A regular expression to match secrets names from the external system. All of
the specified keys must be found on a secret and all of the values must be exact matches. A key with an empty
value, e.g. `""`, must also match an empty value from the external system.
robmonte marked this conversation as resolved.
Show resolved Hide resolved

### Example

In this example, any secret in the GCP source whose name begins with `database/` will be imported into Vault.

```hcl
mapping_regex {
name = "my-map-1"
source = "my-gcp-source-1"
destination = "my-dest-1"
priority = 1
expression = "^database/.*$"
}
```
Loading
Loading