Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Secrets Import documentation #25594

Merged
merged 24 commits into from
Mar 27, 2024
Merged
Show file tree
Hide file tree
Changes from 22 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
b0724f2
Start import docs
robmonte Feb 22, 2024
d793357
Use hideClipboard block on output
robmonte Feb 22, 2024
eaf2a9e
Fix missed line
robmonte Feb 22, 2024
51c9268
Reorganize mappings and source docs, fill out more of the info
robmonte Feb 26, 2024
ce6f30c
Remove copied section
robmonte Feb 28, 2024
22c9a1e
Apply suggestions from code review
robmonte Mar 25, 2024
194a621
Apply suggestions from code review
robmonte Mar 25, 2024
da4323c
Apply suggestions from code review
robmonte Mar 25, 2024
34a5f71
Apply suggestions from code review
robmonte Mar 25, 2024
302831c
Apply suggestions from code review
robmonte Mar 25, 2024
2435ae0
Fix typo
robmonte Mar 25, 2024
9aa1b51
Update website/content/docs/import/index.mdx
robmonte Mar 25, 2024
7af65e8
Apply suggestions from code review
robmonte Mar 25, 2024
11c8ae8
Add experimental partial, apply more suggestions that can't be autoap…
robmonte Mar 25, 2024
f8232dc
Align remaining HCL blocks
robmonte Mar 25, 2024
9735bc7
Fix typos, wording, add links, fill in small bits of info
robmonte Mar 25, 2024
38fb9d7
Update wording and formatting for gcp source and mappings
robmonte Mar 25, 2024
a94fd7c
Merge branch 'main' into docs/secrets-import
robmonte Mar 26, 2024
4e6f242
Change experimental to alpha
robmonte Mar 26, 2024
d14f0e1
Change list tag to alpha
robmonte Mar 26, 2024
2fa7329
Fix file
robmonte Mar 26, 2024
908c858
Apply suggestions from code review
robmonte Mar 26, 2024
5359662
Remove extra line, trigger CI rerun
robmonte Mar 27, 2024
1285247
Merge branch 'main' into docs/secrets-import
robmonte Mar 27, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
103 changes: 103 additions & 0 deletions website/content/docs/commands/operator/import.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
---
layout: docs
page_title: operator import - Command
description: >-
The "operator import" command imports secrets from external systems
in to Vault.
---

# operator import

@include 'alerts/enterprise-only.mdx'

@include 'alerts/alpha.mdx'

The `operator import` command imports secrets from external systems in to Vault.
Secrets with the same name at the same storage path will be overwritten upon import.

<Note title="Imports can be long-running processes">

You can write import plans that read from as many sources as you want. The
amount of data migrated from each source depends on the filters applied and the
dataset available. Be mindful of the time needed to read from each source,
apply any filters, and store the data in Vault.

</Note>

## Examples

Read the config file `import.hcl` to generate a new import plan:

```shell-session
$ vault operator import -config import.hcl plan
```

Output:

<CodeBlockConfig hideClipboard>

-----------
Import plan
-----------
The following namespaces are missing:
* ns-1/

The following mounts are missing:
* ns-1/mount-1

Secrets to be imported to the destination "my-dest-1":
* secret-1
* secret-2

</CodeBlockConfig>

## Configuration

The `operator import` command uses a dedicated configuration file to specify the source,
destination, and mapping rules. To learn more about these types and secrets importing in
general, refer to the [Secrets Import documentation](/vault/docs/import).

```hcl
source_gcp {
name = "my-gcp-source-1"
credentials = "@/path/to/service-account-key.json"
}

destination_vault {
name = "my-dest-1"
address = "http://127.0.0.1:8200/"
token = "root"
namespace = "ns-1"
mount = "mount-1"
}

mapping_passthrough {
name = "my-map-1"
source = "my-gcp-1"
destination = "my-dest-1"
priority = 1
}
```
robmonte marked this conversation as resolved.
Show resolved Hide resolved

## Usage
robmonte marked this conversation as resolved.
Show resolved Hide resolved

### Arguments

- `plan` - Executes a read-only operation to let operators preview the secrets to import based on the configuration file.

- `apply` - Executes the import operations to read the specified secrets from the source and write them into Vault.
Apply first executes a plan, then asks the user to approve the results before performing the actual import.

### Flags

The `operator import` command accepts the following flags:

- `-config` `(string: "import.hcl")` - Path to the import configuration HCL file. The default path is `import.hcl`.

- `-auto-approve` `(bool: <false>)` - Automatically responds "yes" to all user-input prompts for the `apply` command.

- `-auto-create` `(bool: <false>)` - Automatically creates any missing namespaces and KVv2 mounts when
running the `apply` command.

- `-log-level` ((#\_log_level)) `(string: "info")` - Log verbosity level. Supported values (in
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does the # mean?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't actually know, I copied this flag description from another command doc that has it too. There's a handful of files that have this, for example website/content/docs/commands/server.mdx

order of descending detail) are `trace`, `debug`, `info`, `warn`, and `error`. You can also set log-level with the `VAULT_LOG_LEVEL` environment variable.
1 change: 1 addition & 0 deletions website/content/docs/commands/operator/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ Usage: vault operator <subcommand> [options] [args]

Subcommands:
generate-root Generates a new root token
import Import secrets from external systems into Vault
init Initializes a server
key-status Provides information about the active encryption key
rekey Generates new unseal keys
Expand Down
53 changes: 53 additions & 0 deletions website/content/docs/import/gcpsm.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
---
layout: docs
page_title: GCP secret import source
description: The Google Cloud Platform Secret Manager source imports secrets from GCP to Vault.
robmonte marked this conversation as resolved.
Show resolved Hide resolved
---
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The general documentation typically uses a walkthrough style that guides the user through a full-working example from start to finish. Do you think the same pattern would be useful for Secrets Import?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had first copied our usual format based on other docs to base these on, but I started to notice when writing the GCP page that other source pages would be nearly identical. The only section that would be different is explaining the source block with the rest being unchanged. I figure it makes sense to have a small page for each source block, a page for the mappings, and put the destination info on the index page since we are only dealing with Vault as a destination.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Walking folks through a full working example is specific to tutorials. Docs have a more focused purpose. Overviews provide the who/what/where/when/why/how of a feature or function. And how-to guides provide explicit-but-generalized steps for solving specific problems.

If this doc is an overview. Focus on explaining:

  • who can, or would want to, use the feature/functionality
  • what problem the feature/functionality solves
  • where the feature/function is configured or managed
  • when the feature/function is and isn't useful
  • why the feature/function is better than the alternative
  • how it solves the problem at a high level (if a detailed discussion of the the nuts-and-bolts is needed, consider a full technical overview as a separate document)


# GCP import source


Use the GCP source to import secret data from GCP Secret Manager into your Vault instance. To use dynamic
robmonte marked this conversation as resolved.
Show resolved Hide resolved
credentials with GCP import, ensure the [GCP secrets engine](/vault/docs/secrets/gcp) is
already configured.

## Argument reference
robmonte marked this conversation as resolved.
Show resolved Hide resolved

Refer to the [HCL syntax](/vault/docs/import#hcl-syntax-1) for arguments common to all source types.

## Additional arguments

- `credentials` `(string: "")` - The path to the service account key credentials file for the service account
with the [necessary permissions](#permissions). If `credentials` is set, then `vault_mount_path` and
`vault_role_name` must be unset.

- `vault_mount_path` `(string: "")` - The Vault mount path to a pre-configured GCP
secrets engine used to generate dynamic credentials for the importer. If
`vault_mount_path` or `vault_role_name` are set, then `credentials` must be
unset.

- `vault_role_name` `(string: "")` - The Vault role used to generate a dynamic
credential for the importer. The role name must exist in the pre-configured
GCP secrets engine mount. If `vault_role_name` or `vault_mount_path` are set,
then `credentials` must be unset.

## Example

Define and configure the `my-gcp-source-1` GCP source:

```hcl
source_gcp {
name = "my-gcp-source-1"
secrets_engine_mount = "gcp"
secrets_engine_role = "my-gcp-role-1"
}
```

## Permissions

To use GCP import, you must grant the associated GCP identity permission to read secrets:

```shell-session
"secretmanager.secrets.list",
"secretmanager.versions.access",
```
153 changes: 153 additions & 0 deletions website/content/docs/import/index.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,153 @@
---
layout: docs
page_title: Secrets import
description: Secrets import allows you to safely onboard secrets from external sources into Vault KV for management.
---


# Secrets import

@include 'alerts/enterprise-only.mdx'

@include 'alerts/alpha.mdx'

Distributing sensitive information across multiple external systems creates
several challenges, including:

- Increased operational overhead.
- Increased exposure risk from data sprawl.
- Increased risk of outdated and out-of-sync information.

Using Vault as a single source of truth (SSOT) for sensitive data increases
security and reduces management overhead, but migrating preexisting data from multiple
and/or varied sources can be complex and costly.

The secrets import process helps you automate and streamline your sensitive data
migration with codified import plans as HCL files. Import plans tell Vault which KVv2 secrets
engine instance to store the expected secret data in, the source system for which data will be
read from, and how to filter this data. Three HCL blocks make this possible:

- The `destination` block defines target KVv2 mounts.
- The `source` block provides credentials for connecting to the external system.
- The `mapping` block defines how Vault should decide which data gets imported before
writing the information to KVv2.

## Destinations

Vault stores imported secrets in a Vault KVv2 secrets engine mount. Destination
blocks start with `destination_vault` and define the desired KVv2 mount path and
an optional namespace. The combination of these represent the exact location in your
Vault instance you want the information stored.

### HCL syntax


```hcl
destination_vault {
name = "my-dest-1"
namespace = "ns-1"
mount = "mount-1"
}
```

- `name` `(string: <required>)` - A unique name for the destination block that can
be referenced in subsequent mapping blocks.

- `mount` `(string: <required>)` - The mount path for the target KVv2 instance.

- `address` `(string)` - Optional network address of the Vault server with the
KVv2 secrets engine enabled. By default, the Vault client's address will be used.

- `token` `(string)` - Optional authentication token for the Vault server at the
specified address. By default, the Vault client's token will be used.

- `namespace` `(string)` - Optional namespace path containing the specified KVv2
mount. By default, Vault looks for the KVv2 mount under the root namespace.



## Sources
robmonte marked this conversation as resolved.
Show resolved Hide resolved

Vault can import secrets from the following sources:
- [GCP Secret Manager](/vault/docs/import/gcpsm)

To pull data from a source during import, Vault needs read credentials for the
external system. You can provide credentials directly as part of the import
plan, or use Vault to automatically generate dynamic credentials if you already
have the corresponding secrets engine configured.

### HCL syntax

Source blocks start with `source_<external_system>` and include any connection
information required by the target system or the secrets engine to leverage. For example:

```hcl
source_gcp {
name = "my-gcp-source-1"
credentials = "@/path/to/service-account-key.json"
}
```

- `name` `(string: <required>)` - A unique name for the source block that can be
referenced in subsequent mapping blocks.

- `credentials` `(string: <required>)` - Path to a credential file or token with
read permissions for the target system.

Depending on the source system, additional information may be required. Refer to
the connection documentation for your source system to determine the full set of
required fields for that system type.



## Mappings

Mappings glue the source and destination together and filter the migrated data,
to determine what is imported and what is ignored. Vault currently supports the
following mapping methods:

- [mapping_passthrough](/vault/docs/import/mappings#passthrough)
- [mapping_metadata](/vault/docs/import/mappings#metadata)
- [mapping_regex](/vault/docs/import/mappings#regex)

### HCL syntax

Mapping blocks start with `mapping_<filter_type>` and require a source name,
destination name, an execution priority, and any corresponding transformations
or filters that apply for each mapping type. For example:

```hcl
mapping_regex {
name = "my-map-1"
source = "my-gcp-source-1"
destination = "my-dest-1"
priority = 1
expression = "^database/.*$"
}
```

- `name` `(string: <required>)` - A unique name for the mapping block.

- `source` `(string: <required>)` - The name of a previously-defined source block
**from** which the data should be read.

- `destination` `(string: <required>)` - The name of a previously defined
destination block **to** which the data should be written.

- `priority` `(integer: <required>)` - The order in which Vault should apply the
mapping block during the import process. The lower the number, the higher the
priority. For example, a mapping with priority 1 executes before a mapping
with priority 2.

Depending on the filter type, additional fields may be required or possible. Refer
to the [import mappings documentation](/vault/docs/import/mappings) for the available
supported options and for a list of each mapping's specific fields.

<Tip title="Priority matters">

Vault applies mapping definitions in priority order and a given secret only
matches to the first mapping that applies. Once Vault imports a secret with a
particular mapping, subsequent reads from the same source will ignore that
secret. See the [priority section](/vault/docs/import/mappings#priority) for an example.

</Tip>
Loading
Loading