Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add instructions for deploying migration assistant. #8798

Merged
merged 6 commits into from
Dec 2, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
153 changes: 47 additions & 106 deletions _migrations/deploying-migration-assistant/configuration-options.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,66 +7,35 @@

# Configuration options

This page outlines the configuration options for three key migrations:
1. **Metadata Migration**
2. **Backfill Migration with Reindex-from-Snapshot (RFS)**
3. **Live Capture Migration with Capture and Replay (C&R)**
This page outlines the configuration options for three key migrations scenarios:

Each of these migrations may depend on either a snapshot or a capture proxy. The CDK context blocks below are shown as separate context blocks for each migration type for simplicity. If performing multiple migration types, combine these options, as the actual execution of each migration is controlled from the Migration Console.
1. **Metadata migration**
2. **Backfill migration with `Reindex-from-Snapshot` (RFS)**
3. **Live capture migration with Capture and Replay (C&R)**
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the intro file, we do not use C&R as an abbreviation for "Capture and Replay". Please make consistent across files.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Switched to capitalized.


It also has a section describing how to specify the auth details for the source and target cluster (no auth, basic auth with a username and password, or sigv4 auth).
Each of these migrations depend on either a snapshot or a capture proxy. They are shown as AWS Cloud Deployment Kits (CDKs). The CDKs on this page are shown as separate blocks for each migration type. If you are performing a migration that suits multiple scenarios, these options can be combined in the migration console.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

> [!TIP]
For a complete list of configuration options, please refer to the [opensearch-migrations options.md](https://github.com/opensearch-project/opensearch-migrations/blob/main/deployment/cdk/opensearch-service-migration/options.md) but please open an issue for consultation if changing an option that is not listed on this page.

Options for the source cluster endpoint, target cluster endpoint, and existing VPC should be configured for the Migration tools to function effectively.


## Metadata Migration Options
For a complete list of configuration options, please refer to the [opensearch-migrations-options.md](https://github.com/opensearch-project/opensearch-migrations/blob/main/deployment/cdk/opensearch-service-migration/options.md). If there is a configuration option you would like that is not found on this page, open an issue in OpenSearch Migrations repository.

Check warning on line 19 in _migrations/deploying-migration-assistant/configuration-options.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _migrations/deploying-migration-assistant/configuration-options.md#L19

[OpenSearch.Please] Using 'please' is unnecessary. Remove.
Raw output
{"message": "[OpenSearch.Please] Using 'please' is unnecessary. Remove.", "location": {"path": "_migrations/deploying-migration-assistant/configuration-options.md", "range": {"start": {"line": 19, "column": 47}}}, "severity": "WARNING"}

Check failure on line 19 in _migrations/deploying-migration-assistant/configuration-options.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _migrations/deploying-migration-assistant/configuration-options.md#L19

[Vale.Terms] Use 'OpenSearch' instead of 'opensearch'.
Raw output
{"message": "[Vale.Terms] Use 'OpenSearch' instead of 'opensearch'.", "location": {"path": "_migrations/deploying-migration-assistant/configuration-options.md", "range": {"start": {"line": 19, "column": 68}}}, "severity": "ERROR"}
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
{: .tip }
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Line 19: We should probably provide a link to the repo.


## Sample Metadata Migration CDK Options
Options for the source cluster endpoint, target cluster endpoint, and existing VPC should be configured for the Migration tools to function effectively.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

```json
{
"metadata-migration": {
"stage": "dev",
"vpcId": <VPC_ID>,
"sourceCluster": {
"endpoint": <SOURCE_CLUSTER_ENDPOINT>,
"version": "ES 7.10",
"auth": {"type": "none"}
},
"targetCluster": {
"endpoint": <TARGET_CLUSTER_ENDPOINT>,
"auth": {
"type": "basic",
"username": <TARGET_CLUSTER_USERNAME>,
"passwordFromSecretArn": <TARGET_CLUSTER_PASSWORD_SECRET>
}
},
"reindexFromSnapshotServiceEnabled": true,
"artifactBucketRemovalPolicy": "DESTROY"
}
}
```
## Shared configuration options

There are currently no CDK options specific to Metadata migrations, which are performed from the Migration Console. This migration requires an existing snapshot, which can be created from the Migration Console.
Each migration configuration shares the following options.

<details>
<summary><b>Shared configuration options table</b>
</summary>

| Name | Example | Description |
|-----------------------|-----------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `sourceClusterEndpoint` | `"https://source-cluster.elb.us-east-1.endpoint.com"` | The endpoint for the source cluster. |
| `targetClusterEndpoint` | `"https://vpc-demo-opensearch-cluster-cv6hggdb66ybpk4kxssqt6zdhu.us-west-2.es.amazonaws.com:443"` | The endpoint for the target cluster. Required if using an existing target cluster for the migration instead of creating a new one. |
| `vpcId` | `"vpc-123456789abcdefgh"` | The ID of the existing VPC where the migration resources will be placed. The VPC must have at least two private subnets that span two availability zones. |
| Name | Example | Description |
| :--- | :--- | :--- |
| `sourceClusterEndpoint` | `"https://source-cluster.elb.us-east-1.endpoint.com"` | The endpoint for the source cluster. |
| `targetClusterEndpoint` | `"https://vpc-demo-opensearch-cluster-cv6hggdb66ybpk4kxssqt6zdhu.us-west-2.es.amazonaws.com:443"` | The endpoint for the target cluster. Required if using an existing target cluster for the migration instead of creating a new one. |
| `vpcId` | `"vpc-123456789abcdefgh"` | The ID of the existing VPC where the migration resources will be placed. The VPC must have at least two private subnets that span two availability zones. |

Check failure on line 33 in _migrations/deploying-migration-assistant/configuration-options.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _migrations/deploying-migration-assistant/configuration-options.md#L33

[OpenSearch.Spelling] Error: subnets. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.
Raw output
{"message": "[OpenSearch.Spelling] Error: subnets. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_migrations/deploying-migration-assistant/configuration-options.md", "range": {"start": {"line": 33, "column": 227}}}, "severity": "ERROR"}

</details>

## Backfill Migration with Reindex-from-Snapshot (RFS) Options
## Backfill migration using RFS

### Sample Backfill Migration CDK Options
The following CDK performs a backfill migrations using RFS:

```json
{
Expand All @@ -93,22 +62,22 @@
}
```

Performing a Reindex-from-Snapshot backfill migration requires an existing snapshot. The CDK options specific to backfill migrations are listed below. To view all available arguments for `reindexFromSnapshotExtraArgs`, see [here](https://github.com/opensearch-project/opensearch-migrations/blob/main/DocumentsFromSnapshotMigration/README.md#arguments). At a minimum, no extra arguments may be needed.
Performing an RFS backfill migration requires an existing snapshot.

<details>
<summary><b>Backfill specific configuration options table</b>
</summary>

| Name | Example | Description |
|---------------------------------|-----------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `reindexFromSnapshotServiceEnabled` | `true` | Enables deploying and configuring the RFS ECS service. |
| `reindexFromSnapshotExtraArgs` | `"--target-aws-region us-east-1 --target-aws-service-signing-name es"` | Extra arguments for the Document Migration command, with space separation. See the [RFS Extra Arguments](https://github.com/opensearch-project/opensearch-migrations/blob/main/DocumentsFromSnapshotMigration/README.md#arguments) for more details. You can pass `--no-insecure` to remove the `--insecure` flag. |
The RFS CDK uses the following options. To view all available arguments for `reindexFromSnapshotExtraArgs`, see [Snapshot migrations README](https://github.com/opensearch-project/opensearch-migrations/blob/main/DocumentsFromSnapshotMigration/README.md#arguments). At a minimum, no extra arguments may be needed.

Check warning on line 68 in _migrations/deploying-migration-assistant/configuration-options.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _migrations/deploying-migration-assistant/configuration-options.md#L68

[OpenSearch.AcronymParentheses] 'README': Spell out acronyms the first time that you use them on a page and follow them with the acronym in parentheses. Subsequently, use the acronym alone.
Raw output
{"message": "[OpenSearch.AcronymParentheses] 'README': Spell out acronyms the first time that you use them on a page and follow them with the acronym in parentheses. Subsequently, use the acronym alone.", "location": {"path": "_migrations/deploying-migration-assistant/configuration-options.md", "range": {"start": {"line": 68, "column": 243}}}, "severity": "WARNING"}
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

</details>

## Live Capture Migration with Capture and Replay (C&R) Options
| Name | Example | Description |
| :--- | :--- | :--- |
| `reindexFromSnapshotServiceEnabled` | `true` | Enables deploying and configuring the RFS ECS service. |
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
| `reindexFromSnapshotExtraArgs` | `"--target-aws-region us-east-1 --target-aws-service-signing-name es"` | Extra arguments for the Document Migration command, with space separation. See the [RFS Extra Arguments](https://github.com/opensearch-project/opensearch-migrations/blob/main/DocumentsFromSnapshotMigration/README.md#arguments) for more details. You can pass `--no-insecure` to remove the `--insecure` flag. |

Check warning on line 74 in _migrations/deploying-migration-assistant/configuration-options.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _migrations/deploying-migration-assistant/configuration-options.md#L74

[OpenSearch.AcronymParentheses] 'README': Spell out acronyms the first time that you use them on a page and follow them with the acronym in parentheses. Subsequently, use the acronym alone.
Raw output
{"message": "[OpenSearch.AcronymParentheses] 'README': Spell out acronyms the first time that you use them on a page and follow them with the acronym in parentheses. Subsequently, use the acronym alone.", "location": {"path": "_migrations/deploying-migration-assistant/configuration-options.md", "range": {"start": {"line": 74, "column": 315}}}, "severity": "WARNING"}
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

### Sample Live Capture Migration CDK Options
To view all available arguments for `reindexFromSnapshotExtraArgs`, see [Snapshot migrations README](https://github.com/opensearch-project/opensearch-migrations/blob/main/DocumentsFromSnapshotMigration/README.md#arguments). At a minimum, no extra arguments may be needed.

## Live capture migration with C&R

The following sample CDK performs a live capture migration with C&R
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

```json
{
Expand Down Expand Up @@ -137,28 +106,30 @@
}
```

Performing a live capture migration requires that a Capture Proxy be configured to capture incoming traffic and send it to the target cluster via the Traffic Replayer service. For arguments available in `captureProxyExtraArgs`, refer to the `@Parameter` fields [here](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficCaptureProxyServer/src/main/java/org/opensearch/migrations/trafficcapture/proxyserver/CaptureProxy.java). For `trafficReplayerExtraArgs`, refer to the `@Parameter` fields [here](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficReplayer/src/main/java/org/opensearch/migrations/replay/TrafficReplayer.java). At a minimum, no extra arguments may be needed.
Performing a live capture migration requires that a Capture Proxy be configured to capture incoming traffic and send it to the target cluster using the Traffic Replayer service. For arguments available in `captureProxyExtraArgs`, refer to the `@Parameter` fields [here](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficCaptureProxyServer/src/main/java/org/opensearch/migrations/trafficcapture/proxyserver/CaptureProxy.java). For `trafficReplayerExtraArgs`, refer to the `@Parameter` fields [here](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficReplayer/src/main/java/org/opensearch/migrations/replay/TrafficReplayer.java). At a minimum, no extra arguments may be needed.

Check failure on line 109 in _migrations/deploying-migration-assistant/configuration-options.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _migrations/deploying-migration-assistant/configuration-options.md#L109

[OpenSearch.Spelling] Error: Replayer. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.
Raw output
{"message": "[OpenSearch.Spelling] Error: Replayer. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_migrations/deploying-migration-assistant/configuration-options.md", "range": {"start": {"line": 109, "column": 161}}}, "severity": "ERROR"}

<details>
<summary><b>Capture and Replay specific configuration options table</b>
</summary>

| Name | Example | Description |
|--------------------------------|----------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `captureProxyServiceEnabled` | `true` | Enables the Capture Proxy service deployment via a new CloudFormation stack. |
| `captureProxyExtraArgs` | `"--suppressCaptureForHeaderMatch user-agent .*elastic-java/7.17.0.*"` | Extra arguments for the Capture Proxy command, including options specified by the [Capture Proxy](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficCaptureProxyServer/src/main/java/org/opensearch/migrations/trafficcapture/proxyserver/CaptureProxy.java). |
| `trafficReplayerServiceEnabled` | `true` | Enables the Traffic Replayer service deployment via a new CloudFormation stack. |
| Name | Example | Description |
| :--- | :--- | :--- |
| `captureProxyServiceEnabled` | `true` | Enables the Capture Proxy service deployment via a new CloudFormation stack. |

Check warning on line 117 in _migrations/deploying-migration-assistant/configuration-options.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _migrations/deploying-migration-assistant/configuration-options.md#L117

[OpenSearch.LatinismsSubstitution] Use 'using, through, by accessing, or by choosing' instead of 'via'.
Raw output
{"message": "[OpenSearch.LatinismsSubstitution] Use 'using, through, by accessing, or by choosing' instead of 'via'.", "location": {"path": "_migrations/deploying-migration-assistant/configuration-options.md", "range": {"start": {"line": 117, "column": 92}}}, "severity": "WARNING"}
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
| `captureProxyExtraArgs` | `"--suppressCaptureForHeaderMatch user-agent .*elastic-java/7.17.0.*"` | Extra arguments for the Capture Proxy command, including options specified by the [Capture Proxy](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficCaptureProxyServer/src/main/java/org/opensearch/migrations/trafficcapture/proxyserver/CaptureProxy.java). |
| `trafficReplayerServiceEnabled` | `true` | Enables the Traffic Replayer service deployment via a new CloudFormation stack. |

Check failure on line 119 in _migrations/deploying-migration-assistant/configuration-options.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _migrations/deploying-migration-assistant/configuration-options.md#L119

[OpenSearch.Spelling] Error: Replayer. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.
Raw output
{"message": "[OpenSearch.Spelling] Error: Replayer. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_migrations/deploying-migration-assistant/configuration-options.md", "range": {"start": {"line": 119, "column": 67}}}, "severity": "ERROR"}

Check warning on line 119 in _migrations/deploying-migration-assistant/configuration-options.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _migrations/deploying-migration-assistant/configuration-options.md#L119

[OpenSearch.LatinismsSubstitution] Use 'using, through, by accessing, or by choosing' instead of 'via'.
Raw output
{"message": "[OpenSearch.LatinismsSubstitution] Use 'using, through, by accessing, or by choosing' instead of 'via'.", "location": {"path": "_migrations/deploying-migration-assistant/configuration-options.md", "range": {"start": {"line": 119, "column": 95}}}, "severity": "WARNING"}
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
| `trafficReplayerExtraArgs` | `"--sigv4-auth-header-service-region es,us-east-1 --speedup-factor 5"` | Extra arguments for the Traffic Replayer command, including options for auth headers and other parameters specified by the [Traffic Replayer](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficReplayer/src/main/java/org/opensearch/migrations/replay/TrafficReplayer.java). |

</details>

## Cluster Authentication Options
For arguments available in `captureProxyExtraArgs`, refer to the `@Parameter` fields [here](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficCaptureProxyServer/src/main/java/org/opensearch/migrations/trafficcapture/proxyserver/CaptureProxy.java). For `trafficReplayerExtraArgs`, refer to the `@Parameter` fields [here](https://github.com/opensearch-project/opensearch-migrations/blob/main/TrafficCapture/trafficReplayer/src/main/java/org/opensearch/migrations/replay/TrafficReplayer.java). At a minimum, no extra arguments may be needed.


Both the source and target cluster can use no authentication (e.g. limited to the VPC), basic authentication with a username and password, or SigV4 scoped to a user or role.

Examples of each of these are below.
## Cluster authentication pptions

Check failure on line 127 in _migrations/deploying-migration-assistant/configuration-options.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _migrations/deploying-migration-assistant/configuration-options.md#L127

[OpenSearch.Spelling] Error: pptions. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.
Raw output
{"message": "[OpenSearch.Spelling] Error: pptions. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_migrations/deploying-migration-assistant/configuration-options.md", "range": {"start": {"line": 127, "column": 27}}}, "severity": "ERROR"}

Both the source and target cluster can use no authentication, such as authentication limited to VPC, basic authentication with a username and password, or SigV4 scoped to a user or role.

Check warning on line 129 in _migrations/deploying-migration-assistant/configuration-options.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _migrations/deploying-migration-assistant/configuration-options.md#L129

[OpenSearch.SignatureV4] 'Signature Version 4': Use 'AWS Signature Version 4' instead of 'SigV4' on first appearance. Then, Signature Version 4 may be used. Only use SigV4 when space is limited.
Raw output
{"message": "[OpenSearch.SignatureV4] 'Signature Version 4': Use 'AWS Signature Version 4' instead of 'SigV4' on first appearance. Then, Signature Version 4 may be used. Only use SigV4 when space is limited.", "location": {"path": "_migrations/deploying-migration-assistant/configuration-options.md", "range": {"start": {"line": 129, "column": 156}}}, "severity": "WARNING"}
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

### No authentication

No auth:
```
"sourceCluster": {
"endpoint": <SOURCE_CLUSTER_ENDPOINT>,
Expand All @@ -167,7 +138,8 @@
}
```

Basic auth:
### Basic authentication

```
"sourceCluster": {
"endpoint": <SOURCE_CLUSTER_ENDPOINT>,
Expand All @@ -180,7 +152,8 @@
}
```

SigV4 auth:
### SigV4 authentication

Check warning on line 155 in _migrations/deploying-migration-assistant/configuration-options.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _migrations/deploying-migration-assistant/configuration-options.md#L155

[OpenSearch.SignatureV4] 'Signature Version 4': Use 'AWS Signature Version 4' instead of 'SigV4' on first appearance. Then, Signature Version 4 may be used. Only use SigV4 when space is limited.
Raw output
{"message": "[OpenSearch.SignatureV4] 'Signature Version 4': Use 'AWS Signature Version 4' instead of 'SigV4' on first appearance. Then, Signature Version 4 may be used. Only use SigV4 when space is limited.", "location": {"path": "_migrations/deploying-migration-assistant/configuration-options.md", "range": {"start": {"line": 155, "column": 5}}}, "severity": "WARNING"}
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

```
"sourceCluster": {
"endpoint": <SOURCE_CLUSTER_ENDPOINT>,
Expand All @@ -195,40 +168,8 @@

The `serviceSigningName` can be `es` for an Elasticsearch or OpenSearch domain, or `aoss` for an OpenSearch Serverless collection.

All of these auth mechanisms apply to both source and target clusters.

## Troubleshooting

### Restricted Permissions
When deploying if part of an [AWS Organization](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_introduction.html) ↗ some permissions / resources might not be allowed. The full list can be generated from the synthesized cdk output with the awsFeatureUsage script.

```
/opensearch-migrations/deployment/cdk/opensearch-service-migration/awsFeatureUsage.sh [contextId]
```

<details>
<summary><b>Capture and Replay specific configuration options table</b>
</summary>

```shell
$ /opensearch-migrations/deployment/cdk/opensearch-service-migration/awsFeatureUsage.sh default
Synthesizing all stacks...
Synthesizing stack: networkStack-default
Synthesizing stack: migrationInfraStack
Synthesizing stack: reindexFromSnapshotStack
Synthesizing stack: migration-console
Finding resource usage from synthesized stacks...
-----------------------------------
IAM Policy Actions:
cloudwatch:GetMetricData
...
-----------------------------------
Resources Types:
AWS::CDK::Metadata
...
```
</details>
All of these authentication mechanisms apply to both source and target clusters.

## Network configuration

### Network Configuration
The migration tooling expects the source cluster, target cluster, and migration resources to exist in the same VPC. If this is not the case, manual networking setup outside of this documentation is likely required.
Loading
Loading