From a5c70a3bc24de83aa2d4d58a8be80b6a1a7e4d67 Mon Sep 17 00:00:00 2001 From: Archer Date: Thu, 7 Nov 2024 09:09:41 -0600 Subject: [PATCH 01/14] Migration assistant intro Signed-off-by: Archer --- _migrations/Solution-Overview.md | 59 ------------------- _migrations/index.md | 45 ++++++++------ .../is-migration-assistant-right-for-you.md | 43 ++++++++------ 3 files changed, 52 insertions(+), 95 deletions(-) delete mode 100644 _migrations/Solution-Overview.md diff --git a/_migrations/Solution-Overview.md b/_migrations/Solution-Overview.md deleted file mode 100644 index e69b0cd6cf..0000000000 --- a/_migrations/Solution-Overview.md +++ /dev/null @@ -1,59 +0,0 @@ ---- -layout: default -title: Migration assistant overview -nav_order: 15 ---- - -# Migration assistant overview - -This Overview outlines the process for successfully performing an end-to-end migration. The solution offered in this repository caters to several specific scenarios: - -1. Migrating cluster metadata, such as index settings, aliases, and templates. -2. Migrating existing or historical data from a source to a target cluster. -3. Transferring ongoing or live traffic between clusters. -4. Conducting a comprehensive migration involving both existing and live data with zero downtime and the option to back out of a migration. -5. Upgrading an existing cluster. -6. Comparing an existing cluster with a prospective new one. - -In this guide, we focus on scenario 4, guiding you through the migration of historical data from a source cluster while concurrently handling live production traffic, which will be captured and redirected to a target cluster. It's crucial to note that migration strategies are not universally applicable. This guide provides a detailed methodology, based on certain assumptions detailed throughout, emphasizing the importance of robust engineering practices and a systematic approach to ensure a successful migration. - -## Key Components of the Solution - -### Elasticsearch/OpenSearch Source -Your source cluster in this solution operates on Elasticsearch or OpenSearch, hosted on EC2 instances or similar computing environments. A proxy is set up to interact with this source cluster, either positioned in front of or directly on the coordinating nodes of the cluster. - -### Capture Proxy -This component is designed for HTTP RESTful traffic, playing a dual role. It not only forwards traffic to the source cluster but also splits and channels this traffic to a stream-processing service for later playback. - -### Traffic Replayer -Acting as a traffic simulation tool, the Traffic Replayer replays recorded request traffic to a target cluster, mirroring source traffic patterns. It links original requests and their responses to those directed at the target cluster, facilitating comparative analysis. - -### Reindex-from-Snapshort -Reindexing data from an existing snapshot are Elastic Container Service (ECS) workers that coordinate the migration of documents from an existing snapshot, reindexing the documents in parallel to a target cluster. - -### Migration Management Console -A conatiner within Elastic Container Service (ECS) that orchestrates the deployment of the Migration Assistant for Amazon OpenSearch Service, alongside a variety of tools to streamline the migration process. - -### Architecture Overview -The solution architecture, adaptable for cloud deployment, unfolds as follows: - -1. Incoming traffic reaches the existing cluster, targeting each coordinator node. -2. A Capture Proxy is placed before each coordinator node for traffic capture, storing data in an event stream. -3. With the continuous capture setup, historical data backfill is initiated. -4. Post-backfill, the captured traffic is replayed using the Traffic Replayer. -5. The results from directing traffic to both the original and new clusters are then evaluated. - -This architecture is based on the use of AWS cloud infrastructure, but most tools are designed to be cloud-independent. A local containerized version of this solution is also available. - - -Deploying to AWS (covered later in the guide) will deploy the following system design: - -![Migration architecture overview]({{site.url}}{{site.baseurl}}/images/migrations/migration-architecture-overview.svg) - -1. Traffic is directed to the existing cluster. -2. An ALB with Capture Proxies are added allowing for traffic capture and storage in Amazon Managed Streaming for Apache Kafka (MSK). -3. With continuous traffic capture in place, a Reindex-from-Snapshot (RFS) is initiated by the user. -4. Once Reindex-from-Snapshot, traffic captured is replayed by the user using a Traffic Replayer. -5. The user evaluates the outcomes from routing traffic to both the original and the new cluster. -6. After confirming the target cluster’s functionality meets expectations, the user dismantles all related stacks, retaining only the new cluster’s setup. - Additionally, the user may retire and discard the old cluster’s legacy infrastructure. \ No newline at end of file diff --git a/_migrations/index.md b/_migrations/index.md index faddfe41ab..e712a6728e 100644 --- a/_migrations/index.md +++ b/_migrations/index.md @@ -9,43 +9,52 @@ has_toc: false # Migration Assistant for OpenSearch -This overview outlines the process for successfully performing an end-to-end, zero-downtime migration. The solution offered in this repository caters to several specific scenarios: +Migrations Assistant for OpenSearch aids you in successfully performing an end-to-end, zero-downtime migration to OpenSearch from other search providers. It helps with the following scenarios: -1. **Metadata Migration** - Migrating cluster metadata, such as index settings, aliases, and templates. -2. **Backfill Migration** - Migrating existing or historical data from a source to a target cluster. -3. **Live Traffic Migration** - Replicating live ongoing traffic from source to target cluster. -4. **Comparative Tooling** - Comparing the performance and behaviors of an existing cluster with a prospective new one. +- **Metadata migration**: Migrating cluster metadata, such as index settings, aliases, and templates. +- **Backfill migration**: Migrating existing or historical data from a source to a target cluster. +- **Live traffic migration**: Replicating live ongoing traffic from source to target cluster. +- **Comparative tooling**: Comparing the performance and behaviors of an existing cluster with a prospective new one. -In this guide, we focus on scenarios 1-3, guiding you through a backfill from a source cluster while concurrently handling live production traffic, which will be captured and replayed to a target cluster. +This user guide focuses on conducting a comprehensive migration involving both existing and live data with zero downtime and the option to back out of a migration. It's crucial to note that migration strategies are not universally applicable. This guide provides a detailed methodology, based on certain assumptions detailed throughout, emphasizing the importance of robust engineering practices to ensure a successful migration. +{: .tip } +## Key components of Migration Assistant -## Key Components of the Solution +The following are the key components of Migration Assistant. + +### Elasticsearch/OpenSearch source -### Elasticsearch/OpenSearch Source Your source cluster in this solution operates on Elasticsearch or OpenSearch, hosted on EC2 instances or similar computing environments. A proxy is set up to interact with this source cluster, either positioned in front of or directly on the coordinating nodes of the cluster. -### Migration Management Console +### Migration management console + A console that provides a migration-specific CLI and offers a variety of tools to streamline the migration process. Everything necessary for completing a migration, other than cleaning up the migration resources, can be done via this Console. -### Traffic Capture Proxy -This component is designed for HTTP RESTful traffic, playing a dual role. It not only forwards traffic to the source cluster but also splits and channels this traffic to a stream-processing service for later playback. +### Traffic capture proxy + +This component is designed for HTTP RESTful traffic. It forwards traffic to the source cluster and also splits and channels this traffic to a stream-processing service for later playback. ### Traffic Replayer + Acting as a traffic simulation tool, the Traffic Replayer replays recorded request traffic to a target cluster, mirroring source traffic patterns. It links original requests and their responses to those directed at the target cluster, facilitating comparative analysis. -### Metadata Migration Tool -A tool integrated into the Migration CLI that can also be used independently to migrate cluster metadata, including index mappings, index configuration settings, templates, component templates, and aliases. +### Metadata migration tool -### Reindex-from-Snapshot -Reindexing data from an existing snapshot on Elastic Container Service (ECS) workers that coordinate the migration of documents from an existing snapshot, reindexing the documents in parallel to a target cluster. +The Metadata migration tool integrated into the Migration CLI can be used independently to migrate cluster metadata, including index mappings, index configuration settings, templates, component templates, and aliases. -### Target Cluster +### reindex-from-snapshot + +`reindex-from-snapshot` reindexes data from an existing snapshot on Elastic Container Service (ECS) workers that coordinate the migration of documents from an existing snapshot, reindexing the documents in parallel to a target cluster. + +### Target cluster The destination cluster for migration or comparison in an A/B test. -### Architecture Overview -This architecture is based on the use of AWS cloud infrastructure, but most tools are designed to be cloud-independent. A local containerized version of this solution is also available. +### Architecture overview + +The Migration assistant architecture is based on the use of an AWS cloud infrastructure, but most tools are designed to be cloud-independent. A local containerized version of this solution is also available. The design deployed in AWS is as follows: diff --git a/_migrations/is-migration-assistant-right-for-you.md b/_migrations/is-migration-assistant-right-for-you.md index e4a56a435c..e2dadc2766 100644 --- a/_migrations/is-migration-assistant-right-for-you.md +++ b/_migrations/is-migration-assistant-right-for-you.md @@ -1,10 +1,12 @@ --- layout: default -title: Is migration assistant right for you? +title: Is Migration Assistant right for you? nav_order: 5 --- -Before deciding if this is the right tool for you, it's important to assess your specific needs and understand the available tools for performing an upgrade or migration. +# Is Migration Assistant right for you? + +Before deciding if Migration Assistant is right for you, it's important to assess your specific needs and understand the available tools for performing an upgrade or migration. Migration Assistant addresses gaps found in other migration solutions, but in some cases, alternative tools may be a better fit. @@ -24,8 +26,9 @@ There are also tools available for migrating cluster configuration, templates, a | Elasticsearch 7.17 | OpenSearch 2.14 | | OpenSearch 1.3 | OpenSearch 2.14 | -> [!NOTE] -> We expect minor versions within the specified major versions above (i.e., Elasticsearch 6 and 7 and OpenSearch 1 and 2) to be supported, but the versions above are tested. + +Minor versions within the specified major versions above (i.e., Elasticsearch 6 and 7 and OpenSearch 1 and 2) at not yet supported, but the versions above are tested. +{: .note} ### Supported Source and Target Platforms * Self-managed (hosted by cloud provider or on-premises) @@ -33,7 +36,9 @@ There are also tools available for migrating cluster configuration, templates, a The tooling is designed to work with other cloud provider platforms, but it is not officially tested with these other platforms. If you would like to add support, please contact one of the maintainers on [GitHub](https://github.com/opensearch-project/opensearch-migrations/blob/main/MAINTAINERS.md). -### Future Migration Paths +### Future migration paths + +The following table shows planned migration paths: | **Source Version** | **Target Version** | **Tracking Issue** | |-----------------------------|----------------------------------|--------------------| @@ -43,28 +48,30 @@ The tooling is designed to work with other cloud provider platforms, but it is n | Elasitcsearch 1.5 | OpenSearch 2.x |[Issue](https://github.com/opensearch-project/opensearch-migrations/issues/1070)| | OpenSearch 2.x | OpenSearch 2.x |[Issue](https://github.com/opensearch-project/opensearch-migrations/issues/1038)| -## Supported Components +## Supported components Before starting a migration, consider the scope of the components involved. The table below outlines the components that should be considered for migration, indicates their support by the Migration Assistant, and provides comments and recommendations. -| Component | Supported | Comments and Recommendations | -|----------------------------------|---------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| **Documents** | Yes | Migrate existing data with Reindex-from-Snapshot and live traffic with Capture-and-Replay | -| **Index Settings** | Yes | Migrate with Metadata Migration Tool | -| **Index Mappings** | Yes | Migrate with Metadata Migration Tool | -| **Index Templates** | Yes | Migrate with Metadata Migration Tool | -| **Component Templates** | Yes | Migrate with Metadata Migration Tool | +| Component | Supported | Recommendations | +| :--- |:--- | :--- | +| **Documents** | Yes | Migrate existing data with `reindex-from-snapshot` and live traffic with `capture-and-replay`. | +| **Index settings** | Yes | Migrate with the metadata migration tool. | +| **Index mappings** | Yes | Migrate with metadata migration tool. | +| **Index templates** | Yes | Migrate with metadata migration tool. | +| **Component templates** | Yes | Migrate with Metadata Migration Tool | | **Aliases** | Yes | Migrate with Metadata Migration Tool | -| **ISM Policies** | Expected in 2025 | Manually migrate using API | -| **Elasticsearch Kibana Dashboards** | Expected in 2025 | Only needed if tooling is being used to migrate Elasticsearch Kibana Dashboards to OpenSearch Dashboards. Export JSON files from Kibana and import into OpenSearch Dashboards; before importing, use the [dashboardsSanitizer](https://github.com/opensearch-project/opensearch-migrations/tree/main/dashboardsSanitizer) tool on X-Pack visualizations like Canvas and Lens in Kibana Dashboards, as they may require recreation for compatibility with OpenSearch. | -| **Security Constructs** | No | Configure roles and permissions based on cloud provider recommendations. For example, if using AWS, leverage IAM for enhanced security management. | +| **ISM policies** | Expected in 2025 | Manually migrate using API | +| **Elasticsearch kibana dashboards** | Expected in 2025 | Only needed if tooling is being used to migrate Elasticsearch Kibana Dashboards to OpenSearch Dashboards. Export JSON files from Kibana and import into OpenSearch Dashboards; before importing, use the [dashboardsSanitizer](https://github.com/opensearch-project/opensearch-migrations/tree/main/dashboardsSanitizer) tool on X-Pack visualizations like Canvas and Lens in Kibana Dashboards, as they may require recreation for compatibility with OpenSearch. | +| **Security constructs** | No | Configure roles and permissions based on cloud provider recommendations. For example, if using AWS, leverage IAM for enhanced security management. | | **Plugins** | No | Check plugin compatibility; some Elasticsearch plugins may not have direct equivalents in OpenSearch. | --- -## OpenSearch Migrations Assessment - Questionnaire +## Migration Assistant - questionnaire + +The following questionnaire can help you decide whether migration assistant is right for you? -### High-Level / Project Management Questions: +### High-level / Project management questions: 1. **By when do you expect the migration to be completed?** *Why it’s important:* From bf441d35ed19795612f28efa76ef1d343fcb648b Mon Sep 17 00:00:00 2001 From: Archer Date: Thu, 7 Nov 2024 09:15:17 -0600 Subject: [PATCH 02/14] Edit solutions architecture overview. Signed-off-by: Archer --- _migrations/index.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/_migrations/index.md b/_migrations/index.md index e712a6728e..9164446d92 100644 --- a/_migrations/index.md +++ b/_migrations/index.md @@ -61,8 +61,8 @@ The design deployed in AWS is as follows: ![Migration architecture overview]({{site.url}}{{site.baseurl}}/images/migrations/migration-architecture-overview.svg) 1. Client traffic is directed to the existing cluster. -2. An ALB with Capture Proxies relaying traffic to source while replicating to Amazon MSK. -3. With continuous traffic capture in place, a Reindex-from-Snapshot (RFS) is initiated by the user via Migration Console. -4. Once Reindex-from-Snapshot is complete, traffic captured is replayed from MSK by Traffic Replayer. -5. Performance and behavior of traffic sent to source and target clusters are compared by reviewing logs and metrics. -6. After confirming the target cluster’s functionality meets expectations the use redirects clients to new target. +2. An Application Load Balancer (ALB) with capture proxies relays traffic to a source while replicating data to Amazon Managed Streaming for Apace Kafka (AWS MSK). +3. With continuous traffic capture in place, a `reindex-from-snapshot` is initiated by the user via Migration Console. +4. Once `reindex-from-snapshot` is complete, traffic captured is replayed from AWS MSK by Traffic Replayer. +5. Performance and behavior of traffic sent to the source and target clusters are compared by reviewing logs and metrics. +6. After confirming the target cluster’s functionality meets expectations, clients are redirected to the new target. From 5a558212073f2e9b266fa0620c3af8d15a7762e2 Mon Sep 17 00:00:00 2001 From: Archer Date: Thu, 7 Nov 2024 09:27:03 -0600 Subject: [PATCH 03/14] FAQ updates. Signed-off-by: Archer --- _migrations/index.md | 1 + .../is-migration-assistant-right-for-you.md | 35 +++++++++++-------- 2 files changed, 21 insertions(+), 15 deletions(-) diff --git a/_migrations/index.md b/_migrations/index.md index 9164446d92..898376a41c 100644 --- a/_migrations/index.md +++ b/_migrations/index.md @@ -50,6 +50,7 @@ The Metadata migration tool integrated into the Migration CLI can be used indepe `reindex-from-snapshot` reindexes data from an existing snapshot on Elastic Container Service (ECS) workers that coordinate the migration of documents from an existing snapshot, reindexing the documents in parallel to a target cluster. ### Target cluster + The destination cluster for migration or comparison in an A/B test. ### Architecture overview diff --git a/_migrations/is-migration-assistant-right-for-you.md b/_migrations/is-migration-assistant-right-for-you.md index e2dadc2766..12820ff188 100644 --- a/_migrations/is-migration-assistant-right-for-you.md +++ b/_migrations/is-migration-assistant-right-for-you.md @@ -14,7 +14,7 @@ For instance, if you need to upgrade more than one major version—such as movin There are also tools available for migrating cluster configuration, templates, and aliases, which can be helpful depending on the complexity of your setup. These tools streamline the migration process by preserving critical settings and custom configurations. -## Migration Paths +## Migration paths | **Source Version** | **Target Version** | |-----------------------------|----------------------------------| @@ -75,11 +75,11 @@ The following questionnaire can help you decide whether migration assistant is r 1. **By when do you expect the migration to be completed?** *Why it’s important:* - Understanding the timeline helps set realistic expectations and plan resources accordingly. + Understanding the timeline helps set realistic expectations for your migration and allows you to plan resources accordingly. 2. **How much downtime can you afford during the migration?** *Why it’s important:* - This defines the migration strategy, whether zero-downtime or with controlled outages, and helps prevent unexpected service disruptions. + This defines a migration strategy that helps prevent unexpected service disruptions. Decide whether you can afford zero-downtime or migrate with controlled outages. 3. **What are your primary goals for the migration?** (e.g., Performance improvement, cost reduction, feature enhancement, etc.) @@ -87,29 +87,33 @@ The following questionnaire can help you decide whether migration assistant is r Clarifying the objectives helps tailor the migration process to meet the customer’s expectations and success criteria. 4. **What's your level of acceptance for discrepancies during the upgrade?** - (e.g., If relevancy is re-ordered because the two clusters aren’t exactly in sync, is this tolerable?) + ( If relevancy is re-ordered because the two clusters aren’t exactly in sync, is this tolerable?) *Why it’s important:* - Helps gauge the flexibility of the customer, especially when dealing with non-deterministic elements like ranking in search results or latency variations. + + For example, decide whether if relevancy is re-ordered because the two clusters aren’t exactly in sync, is this tolerable? This will help gauge the flexibility of the customer, especially when dealing with non-deterministic elements like ranking in search results or latency variations. 5. **Are there any SLAs in place for downstream users or services?** *Why it’s important:* - Understanding SLAs ensures that any changes to performance or downtime do not violate contractual agreements, maintaining service quality. + + Understanding what SLAs are currently in place ensures that any changes to performance or downtime does not violate contractual agreements, maintaining service quality. 6. **Do you foresee any regulatory or compliance constraints that might impact the migration timeline or execution?** *Why it’s important:* + Compliance requirements can dictate migration approaches, particularly regarding data handling, encryption, or audit logging. 7. **Are you open to a trusted proxy preceding your source cluster, even if it adds a 10-30ms latency?** *Why it’s important:* - A proxy can make certain migrations smoother (e.g., for reindexing or log collection), but it’s crucial to assess the customer’s tolerance for added latency. + + A proxy can make certain migrations smoother, like for reindexing or log collection, but it’s crucial to assess the customer’s tolerance for added latency. 8. **Do you possess a dedicated environment for trial upgrades before the production shift?** - (Please outline your typical upgrade process.) *Why it’s important:* - Testing in a dedicated environment before production can prevent unexpected issues and allow for smoother transitions. + Outline what you typical upgrade process looks like. Testing in a dedicated environment before production can prevent unexpected issues and allow for smoother transitions. 9. **Do you have a rollback plan in case the migration encounters issues?** *Why it’s important:* + Rollback strategies help ensure business continuity in case the migration fails, minimizing risks to the business. --- @@ -119,30 +123,31 @@ The following questionnaire can help you decide whether migration assistant is r #### Platform & Architecture: 1. **What are the source and target platforms and their versions?** - (e.g., Elasticsearch 7.10.2 to OpenSearch 2.x) *Why it’s important:* - Compatibility between versions is key for smooth migrations, ensuring the right tools and processes are used. + [Compatibility between versions](#migration-paths) is key for smooth migrations, ensuring the right tools and processes are used. 2. **Provide a breakdown of nodes in the cluster.** (e.g., Total nodes, number of coordinating nodes, and number of data nodes.) *Why it’s important:* - The structure of the cluster affects the complexity of the migration and helps in resource planning. + Determine the total number of nodes in your cluster, including the number of coordinating nodes and the number of data nodes. The structure of the cluster affects the complexity of the migration and helps in resource planning. -3. **Are retention policies in place?** - (If so, for how long, and is it on all indices?) +3. **Are retention policies in place?** *Why it’s important:* - Retention policies could influence the size of the data that needs to be migrated and how backups or archiving strategies are handled. They may also impact the recommended migration solution. + If retention policies are in place, how long will those policies last and are they relevant to all indexes. Retention policies could influence the size of the data that needs to be migrated and how backups or archiving strategies are handled. They may also impact the recommended migration solution. 4. **How do you currently handle backup and recovery in your cluster?** *Why it’s important:* + A solid backup and recovery strategy is essential for minimizing data loss and ensuring a smooth migration. 5. **Are there schematic diagrams showcasing how Elasticsearch/OpenSearch integrates with your entire system?** *Why it’s important:* + Understanding the full architecture helps ensure that the migration doesn’t break any dependencies and that integrations continue to work post-migration. 6. **Which authentication and authorization mechanisms are in place?** *Why it’s important:* + Security configurations need to be migrated and verified to maintain data access controls and compliance post-migration. --- From 3168c2220e17212448f9dc9e086813011a221b93 Mon Sep 17 00:00:00 2001 From: Archer Date: Thu, 7 Nov 2024 09:43:06 -0600 Subject: [PATCH 04/14] Add additional edits. Signed-off-by: Archer --- _migrations/is-migration-assistant-right-for-you.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/_migrations/is-migration-assistant-right-for-you.md b/_migrations/is-migration-assistant-right-for-you.md index 12820ff188..234ca663a5 100644 --- a/_migrations/is-migration-assistant-right-for-you.md +++ b/_migrations/is-migration-assistant-right-for-you.md @@ -147,7 +147,7 @@ The following questionnaire can help you decide whether migration assistant is r 6. **Which authentication and authorization mechanisms are in place?** *Why it’s important:* - + Security configurations need to be migrated and verified to maintain data access controls and compliance post-migration. --- @@ -181,7 +181,7 @@ The following questionnaire can help you decide whether migration assistant is r --- -#### Application & Use: +#### Application & use 1. **What's the main application of your cluster?** (e.g., search, logging, analytics, etc.) From 1c5f86dec39078cb0d37f673cc81ba68965f710c Mon Sep 17 00:00:00 2001 From: Archer Date: Mon, 11 Nov 2024 17:25:45 -0600 Subject: [PATCH 05/14] Add addtional edits. Signed-off-by: Archer --- .../is-migration-assistant-right-for-you.md | 210 +----------------- _migrations/meta.md | 26 --- 2 files changed, 9 insertions(+), 227 deletions(-) delete mode 100644 _migrations/meta.md diff --git a/_migrations/is-migration-assistant-right-for-you.md b/_migrations/is-migration-assistant-right-for-you.md index 234ca663a5..927de6abaf 100644 --- a/_migrations/is-migration-assistant-right-for-you.md +++ b/_migrations/is-migration-assistant-right-for-you.md @@ -19,18 +19,18 @@ There are also tools available for migrating cluster configuration, templates, a | **Source Version** | **Target Version** | |-----------------------------|----------------------------------| | Elasticsearch 6.8 | OpenSearch 1.3 | -| Elasticsearch 6.8 | OpenSearch 2.14 | +| Elasticsearch 6.8 | OpenSearch 2.15 | | Elasticsearch 7.10.2 | OpenSearch 1.3 | -| Elasticsearch 7.10.2 | OpenSearch 2.14 | +| Elasticsearch 7.10.2 | OpenSearch 2.15 | | Elasticsearch 7.17 | OpenSearch 1.3 | -| Elasticsearch 7.17 | OpenSearch 2.14 | -| OpenSearch 1.3 | OpenSearch 2.14 | +| Elasticsearch 7.17 | OpenSearch 2.15 | +| OpenSearch 1.3 | OpenSearch 2.15 | Minor versions within the specified major versions above (i.e., Elasticsearch 6 and 7 and OpenSearch 1 and 2) at not yet supported, but the versions above are tested. {: .note} -### Supported Source and Target Platforms +### Supported source and target platforms * Self-managed (hosted by cloud provider or on-premises) * AWS OpenSearch @@ -38,15 +38,7 @@ The tooling is designed to work with other cloud provider platforms, but it is n ### Future migration paths -The following table shows planned migration paths: - -| **Source Version** | **Target Version** | **Tracking Issue** | -|-----------------------------|----------------------------------|--------------------| -| Elasticsearch 8.x | OpenSearch 2.x |[Issue](https://github.com/opensearch-project/opensearch-migrations/issues/1071)| -| Elasticsearch 5.6 | OpenSearch 2.x |[Issue](https://github.com/opensearch-project/opensearch-migrations/issues/1067)| -| Elasticsearch 2.3 | OpenSearch 2.x |[Issue](https://github.com/opensearch-project/opensearch-migrations/issues/1069)| -| Elasitcsearch 1.5 | OpenSearch 2.x |[Issue](https://github.com/opensearch-project/opensearch-migrations/issues/1070)| -| OpenSearch 2.x | OpenSearch 2.x |[Issue](https://github.com/opensearch-project/opensearch-migrations/issues/1038)| +To see the roadmap for OpenSearch migrations, go to [OpenSarch Migration - Roadmap](https://github.com/orgs/opensearch-project/projects/229/views/1). ## Supported components @@ -54,7 +46,7 @@ Before starting a migration, consider the scope of the components involved. The | Component | Supported | Recommendations | | :--- |:--- | :--- | -| **Documents** | Yes | Migrate existing data with `reindex-from-snapshot` and live traffic with `capture-and-replay`. | +| **Documents** | Yes | Migrate existing data with Reindex from Snapshot (RFS) and live traffic with Capture and Replay. | | **Index settings** | Yes | Migrate with the metadata migration tool. | | **Index mappings** | Yes | Migrate with metadata migration tool. | | **Index templates** | Yes | Migrate with metadata migration tool. | @@ -65,192 +57,8 @@ Before starting a migration, consider the scope of the components involved. The | **Security constructs** | No | Configure roles and permissions based on cloud provider recommendations. For example, if using AWS, leverage IAM for enhanced security management. | | **Plugins** | No | Check plugin compatibility; some Elasticsearch plugins may not have direct equivalents in OpenSearch. | ---- - -## Migration Assistant - questionnaire - -The following questionnaire can help you decide whether migration assistant is right for you? - -### High-level / Project management questions: - -1. **By when do you expect the migration to be completed?** - *Why it’s important:* - Understanding the timeline helps set realistic expectations for your migration and allows you to plan resources accordingly. - -2. **How much downtime can you afford during the migration?** - *Why it’s important:* - This defines a migration strategy that helps prevent unexpected service disruptions. Decide whether you can afford zero-downtime or migrate with controlled outages. - -3. **What are your primary goals for the migration?** - (e.g., Performance improvement, cost reduction, feature enhancement, etc.) - *Why it’s important:* - Clarifying the objectives helps tailor the migration process to meet the customer’s expectations and success criteria. - -4. **What's your level of acceptance for discrepancies during the upgrade?** - ( If relevancy is re-ordered because the two clusters aren’t exactly in sync, is this tolerable?) - *Why it’s important:* - - For example, decide whether if relevancy is re-ordered because the two clusters aren’t exactly in sync, is this tolerable? This will help gauge the flexibility of the customer, especially when dealing with non-deterministic elements like ranking in search results or latency variations. - -5. **Are there any SLAs in place for downstream users or services?** - *Why it’s important:* - - Understanding what SLAs are currently in place ensures that any changes to performance or downtime does not violate contractual agreements, maintaining service quality. - -6. **Do you foresee any regulatory or compliance constraints that might impact the migration timeline or execution?** - *Why it’s important:* - - Compliance requirements can dictate migration approaches, particularly regarding data handling, encryption, or audit logging. - -7. **Are you open to a trusted proxy preceding your source cluster, even if it adds a 10-30ms latency?** - *Why it’s important:* - - A proxy can make certain migrations smoother, like for reindexing or log collection, but it’s crucial to assess the customer’s tolerance for added latency. - -8. **Do you possess a dedicated environment for trial upgrades before the production shift?** - *Why it’s important:* - Outline what you typical upgrade process looks like. Testing in a dedicated environment before production can prevent unexpected issues and allow for smoother transitions. - -9. **Do you have a rollback plan in case the migration encounters issues?** - *Why it’s important:* - - Rollback strategies help ensure business continuity in case the migration fails, minimizing risks to the business. - ---- - -### Technical Questions - -#### Platform & Architecture: - -1. **What are the source and target platforms and their versions?** - *Why it’s important:* - [Compatibility between versions](#migration-paths) is key for smooth migrations, ensuring the right tools and processes are used. - -2. **Provide a breakdown of nodes in the cluster.** - (e.g., Total nodes, number of coordinating nodes, and number of data nodes.) - *Why it’s important:* - Determine the total number of nodes in your cluster, including the number of coordinating nodes and the number of data nodes. The structure of the cluster affects the complexity of the migration and helps in resource planning. - -3. **Are retention policies in place?** - *Why it’s important:* - If retention policies are in place, how long will those policies last and are they relevant to all indexes. Retention policies could influence the size of the data that needs to be migrated and how backups or archiving strategies are handled. They may also impact the recommended migration solution. - -4. **How do you currently handle backup and recovery in your cluster?** - *Why it’s important:* - - A solid backup and recovery strategy is essential for minimizing data loss and ensuring a smooth migration. - -5. **Are there schematic diagrams showcasing how Elasticsearch/OpenSearch integrates with your entire system?** - *Why it’s important:* - - Understanding the full architecture helps ensure that the migration doesn’t break any dependencies and that integrations continue to work post-migration. - -6. **Which authentication and authorization mechanisms are in place?** - *Why it’s important:* - - Security configurations need to be migrated and verified to maintain data access controls and compliance post-migration. - ---- - -#### Data & Cluster Metrics: - -1. **How much data is stored in the cluster?** - *Why it’s important:* - Helps estimate the time and resources required for data transfer, and determines the migration method (batch or streaming). - -2. **Could you specify the mean and peak throughput of the cluster?** - *Why it’s important:* - Understanding the performance requirements ensures that the target environment can handle both normal and peak loads. - -3. **What are the transactions per second?** - (Also, indices per second, queries per second.) - *Why it’s important:* - This allows for proper sizing and performance tuning of the target cluster. - -4. **How many shards do you have and what is the max shard size?** - *Why it’s important:* - Shard size and distribution affect performance and the ease of migration, especially in large datasets. - -5. **What version of the Lucene index are you currently using?** - *Why it’s important:* - Index compatibility is critical to ensure the integrity of the data when moving between different versions of Elasticsearch or OpenSearch. - -6. **Do source documents contain a _source field?** - *Why it’s important:* - The _source field is essential for data recovery and reindexing; knowing this helps in planning for data replication or migration methods. - ---- - -#### Application & use - -1. **What's the main application of your cluster?** - (e.g., search, logging, analytics, etc.) - *Why it’s important:* - The use case determines the criticality of different features (e.g., search accuracy vs. logging performance) in the target environment. - -2. **How many client applications connect to the cluster?** - (Are you aware of any potential upgrade compatibility concerns?) - *Why it’s important:* - Compatibility with clients is crucial for a smooth migration. Some clients might need reconfiguration or updates. - -3. **Are there any specific integrations (e.g., AWS, GCP, third-party services) that need to be tested during the migration?** - *Why it’s important:* - Identifying critical integrations helps ensure they continue to function post-migration, preventing system-wide issues. - -4. **Share any specifics about the requests directed at your cluster that might pose migration challenges.** - (e.g., request format, payload size.) - *Why it’s important:* - Helps identify any request patterns or payloads that might be problematic during the migration process. - -5. **Do clients hit source domains directly, or is there a layer of indirection?** - (e.g., load balancer) - *Why it’s important:* - Understanding how traffic flows into the cluster helps in designing a migration plan with minimal client disruption. - ---- - -#### Security & Plugins: - -1. **Is inter-node encryption active within the cluster?** - *Why it’s important:* - Ensures that the migration process adheres to the customer’s security requirements, especially for sensitive data. - -2. **Do you have data encryption at rest enabled?** - *Why it’s important:* - Encryption settings might need reconfiguration post-migration, and this helps avoid compliance or security issues. - -3. **List any plugins in use that will require migration.** - *Why it’s important:* - Some plugins may not be compatible with newer versions of OpenSearch, requiring alternatives or upgrades. - -4. **Are you utilizing Kibana or any other visualization tools?** - *Why it’s important:* - Ensures that dashboards and visualization setups are preserved or replicated post-migration. - -5. **Are you aware of any custom scripts or ingest pipelines currently in use?** - *Why it’s important:* - Custom components may require validation or modification during the migration, preventing post-migration breakage. - ---- - -#### History & Deployment: - -1. **Provide the details of the most recent upgrade, including versions involved.** - (If available, a history of all upgrade paths would be beneficial.) - *Why it’s important:* - Helps in assessing potential issues with upgrade compatibility and identifies if any manual fixes were applied during previous upgrades. - -2. **How do you install/deploy Elasticsearch/OpenSearch?** - *Why it’s important:* - Deployment methods (e.g., Docker, Kubernetes, bare metal) will affect the migration process and the setup of the target cluster. -3. **Do you have any custom monitoring or alerting tools set up for the cluster?** - (e.g., Datadog, Prometheus, OpenSearch Dashboards.) - *Why it’s important:* - Monitoring tools may require reconfiguration to work with the new environment, ensuring continued operational visibility. -4. **Can you provide the original version of Elasticsearch/OpenSearch that was used for each of the indices in your cluster?** - *Why it’s important:* - The version history of indices is critical because older index versions may not be compatible with the current or target platform during the migration. Knowing the original versions helps identify any necessary reindexing steps, ensures compatibility with the target cluster, and prevents potential issues related to index corruption, feature deprecation, or incompatibility with newer versions of OpenSearch. It also aids in planning for adjustments in index mappings and settings that may have changed over different versions. +## Migration assessment questionnaire -These same questions are available in +If you're having trouble deciding whether migration [OpenSearch Migrations Assessment - Version 5.docx](https://github.com/user-attachments/files/17177220/OpenSearch.Migrations.Assessment.-.Version.5.docx). \ No newline at end of file diff --git a/_migrations/meta.md b/_migrations/meta.md deleted file mode 100644 index 87e763b847..0000000000 --- a/_migrations/meta.md +++ /dev/null @@ -1,26 +0,0 @@ - -This page is meant to describe how the Migration Assistant (MA) documentation is organization and used. - -## Organization -1. All pages should have a commented out section with metadata information used for determining what styles should be applied, eg. ``. - - **Meta** these follow unique conventions to the documentation's implementation. - - **Home** This is the primary entry point to the project, there should only be one such page. - - **Guide** take a user through an substantial part of the MA workflow. Duplicated information is OK. They should be styled simaliarally to one another. - - **Topic** describe a specific area of functionality in MA and are meant to be digestible from top to bottom in a single sitting. After the user has read the topic they would use a sidebar navigate to pick the next topic. -2. Sidebar navigation should include all guides and topics. -3. There is no hierarchy in project all files are off of the root. -4. All internal links should use the `[[|]]` format so testing on forks is easy and broken page links are easier to detect. - -## Style - -### Links -1. Links to any external documentation that isn't directly under the purview of the migrations team should be followed with a space and the `↗` character, eg. `[link](...) ↗`. - -### Format -1. Only the home page is has level 1 `#` headers, all other pages should start with a paragraph and the first header start at `##`. -2. Standard sections - - `## Troubleshooting` - 1. No summary should be included. - 2. Call out a clear problem or resolution, eg. `### Slow Snapshot Speed`. - - `## Related Links` - - All links should be bulleted and labeled `- [Migration Console ALB Documentation](...)` From 475be810572add006b48663bed9886710cf44c97 Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Fri, 15 Nov 2024 11:32:16 -0600 Subject: [PATCH 06/14] Apply suggestions from code review Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- _migrations/is-migration-assistant-right-for-you.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/_migrations/is-migration-assistant-right-for-you.md b/_migrations/is-migration-assistant-right-for-you.md index 927de6abaf..446c925ccd 100644 --- a/_migrations/is-migration-assistant-right-for-you.md +++ b/_migrations/is-migration-assistant-right-for-you.md @@ -27,7 +27,6 @@ There are also tools available for migrating cluster configuration, templates, a | OpenSearch 1.3 | OpenSearch 2.15 | -Minor versions within the specified major versions above (i.e., Elasticsearch 6 and 7 and OpenSearch 1 and 2) at not yet supported, but the versions above are tested. {: .note} ### Supported source and target platforms @@ -53,7 +52,7 @@ Before starting a migration, consider the scope of the components involved. The | **Component templates** | Yes | Migrate with Metadata Migration Tool | | **Aliases** | Yes | Migrate with Metadata Migration Tool | | **ISM policies** | Expected in 2025 | Manually migrate using API | -| **Elasticsearch kibana dashboards** | Expected in 2025 | Only needed if tooling is being used to migrate Elasticsearch Kibana Dashboards to OpenSearch Dashboards. Export JSON files from Kibana and import into OpenSearch Dashboards; before importing, use the [dashboardsSanitizer](https://github.com/opensearch-project/opensearch-migrations/tree/main/dashboardsSanitizer) tool on X-Pack visualizations like Canvas and Lens in Kibana Dashboards, as they may require recreation for compatibility with OpenSearch. | +| **Elasticsearch Kibana dashboards** | Expected in 2025 | Only needed if tooling is being used to migrate Elasticsearch Kibana Dashboards to OpenSearch Dashboards. Export JSON files from Kibana and import into OpenSearch Dashboards; before importing, use the [`dashboardsSanitizer`](https://github.com/opensearch-project/opensearch-migrations/tree/main/dashboardsSanitizer) tool on X-Pack visualizations like Canvas and Lens in Kibana Dashboards, as they may require recreation for compatibility with OpenSearch. | | **Security constructs** | No | Configure roles and permissions based on cloud provider recommendations. For example, if using AWS, leverage IAM for enhanced security management. | | **Plugins** | No | Check plugin compatibility; some Elasticsearch plugins may not have direct equivalents in OpenSearch. | From cb86d5c98d288af754442f3bd97804a2da36dd7d Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Fri, 15 Nov 2024 11:34:24 -0600 Subject: [PATCH 07/14] Apply suggestions from code review Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- _migrations/is-migration-assistant-right-for-you.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_migrations/is-migration-assistant-right-for-you.md b/_migrations/is-migration-assistant-right-for-you.md index 446c925ccd..caff1615e4 100644 --- a/_migrations/is-migration-assistant-right-for-you.md +++ b/_migrations/is-migration-assistant-right-for-you.md @@ -52,7 +52,7 @@ Before starting a migration, consider the scope of the components involved. The | **Component templates** | Yes | Migrate with Metadata Migration Tool | | **Aliases** | Yes | Migrate with Metadata Migration Tool | | **ISM policies** | Expected in 2025 | Manually migrate using API | -| **Elasticsearch Kibana dashboards** | Expected in 2025 | Only needed if tooling is being used to migrate Elasticsearch Kibana Dashboards to OpenSearch Dashboards. Export JSON files from Kibana and import into OpenSearch Dashboards; before importing, use the [`dashboardsSanitizer`](https://github.com/opensearch-project/opensearch-migrations/tree/main/dashboardsSanitizer) tool on X-Pack visualizations like Canvas and Lens in Kibana Dashboards, as they may require recreation for compatibility with OpenSearch. | +| **Elasticsearch Kibana dashboards** | Expected in 2025 | This tool is only needed when used to migrate Elasticsearch Kibana Dashboards to OpenSearch Dashboards. To start, export JSON files from Kibana and import into OpenSearch Dashboards; before importing, use the [`dashboardsSanitizer`](https://github.com/opensearch-project/opensearch-migrations/tree/main/dashboardsSanitizer) tool on X-Pack visualizations like Canvas and Lens in Kibana Dashboards, as they may require recreation for compatibility with OpenSearch. | | **Security constructs** | No | Configure roles and permissions based on cloud provider recommendations. For example, if using AWS, leverage IAM for enhanced security management. | | **Plugins** | No | Check plugin compatibility; some Elasticsearch plugins may not have direct equivalents in OpenSearch. | From e5411750077873373370a311397b27fdf4f7c3e1 Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Mon, 18 Nov 2024 15:17:48 -0600 Subject: [PATCH 08/14] Apply suggestions from code review Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- _migrations/index.md | 2 +- _migrations/is-migration-assistant-right-for-you.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/_migrations/index.md b/_migrations/index.md index 898376a41c..0bf4327e05 100644 --- a/_migrations/index.md +++ b/_migrations/index.md @@ -64,6 +64,6 @@ The design deployed in AWS is as follows: 1. Client traffic is directed to the existing cluster. 2. An Application Load Balancer (ALB) with capture proxies relays traffic to a source while replicating data to Amazon Managed Streaming for Apace Kafka (AWS MSK). 3. With continuous traffic capture in place, a `reindex-from-snapshot` is initiated by the user via Migration Console. -4. Once `reindex-from-snapshot` is complete, traffic captured is replayed from AWS MSK by Traffic Replayer. +4. Once `reindex-from-snapshot` is complete, captured traffic is replayed from AWS MSK to the target cluster by the traffic replayer. 5. Performance and behavior of traffic sent to the source and target clusters are compared by reviewing logs and metrics. 6. After confirming the target cluster’s functionality meets expectations, clients are redirected to the new target. diff --git a/_migrations/is-migration-assistant-right-for-you.md b/_migrations/is-migration-assistant-right-for-you.md index caff1615e4..a5a0fce958 100644 --- a/_migrations/is-migration-assistant-right-for-you.md +++ b/_migrations/is-migration-assistant-right-for-you.md @@ -10,7 +10,7 @@ Before deciding if Migration Assistant is right for you, it's important to asses Migration Assistant addresses gaps found in other migration solutions, but in some cases, alternative tools may be a better fit. -For instance, if you need to upgrade more than one major version—such as moving from Elasticsearch 6.8 to OpenSearch 2.3—Migration Assistant allows you to do this in a single hop. In contrast, other options like rolling upgrades or snapshot restore would require multiple steps, as they cannot handle major version jumps without reindexing your data. Additionally, if you need to capture live traffic and perform a zero-downtime migration, Migration Assistant would be the right choice. +For instance, if you need to upgrade more than one major version, such as moving from Elasticsearch 6.8 to OpenSearch 2.15, Migration Assistant allows you to do this in a single hop. In contrast, other options like rolling upgrades or snapshot restore would require multiple steps, as they cannot handle major version jumps without reindexing your data. Additionally, if you need to capture live traffic and perform a zero-downtime migration, Migration Assistant would be the right choice. There are also tools available for migrating cluster configuration, templates, and aliases, which can be helpful depending on the complexity of your setup. These tools streamline the migration process by preserving critical settings and custom configurations. @@ -50,7 +50,7 @@ Before starting a migration, consider the scope of the components involved. The | **Index mappings** | Yes | Migrate with metadata migration tool. | | **Index templates** | Yes | Migrate with metadata migration tool. | | **Component templates** | Yes | Migrate with Metadata Migration Tool | -| **Aliases** | Yes | Migrate with Metadata Migration Tool | +| **Aliases** | Yes | Migrate with metadata migration tool | | **ISM policies** | Expected in 2025 | Manually migrate using API | | **Elasticsearch Kibana dashboards** | Expected in 2025 | This tool is only needed when used to migrate Elasticsearch Kibana Dashboards to OpenSearch Dashboards. To start, export JSON files from Kibana and import into OpenSearch Dashboards; before importing, use the [`dashboardsSanitizer`](https://github.com/opensearch-project/opensearch-migrations/tree/main/dashboardsSanitizer) tool on X-Pack visualizations like Canvas and Lens in Kibana Dashboards, as they may require recreation for compatibility with OpenSearch. | | **Security constructs** | No | Configure roles and permissions based on cloud provider recommendations. For example, if using AWS, leverage IAM for enhanced security management. | From a3eb9ef9361340ad924ce2f1a1a612fc1b36fbde Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Thu, 21 Nov 2024 14:50:04 -0600 Subject: [PATCH 09/14] Update _migrations/index.md Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- _migrations/index.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/_migrations/index.md b/_migrations/index.md index 0bf4327e05..b4eed54074 100644 --- a/_migrations/index.md +++ b/_migrations/index.md @@ -63,7 +63,8 @@ The design deployed in AWS is as follows: 1. Client traffic is directed to the existing cluster. 2. An Application Load Balancer (ALB) with capture proxies relays traffic to a source while replicating data to Amazon Managed Streaming for Apace Kafka (AWS MSK). -3. With continuous traffic capture in place, a `reindex-from-snapshot` is initiated by the user via Migration Console. +3. Using the migration console, you can initiate metadata migration to establish indexes, templates, component templates, and aliases on the target cluster. +4. With continuous traffic capture in place, you can use a `reindex-from-snapshot` process to capture data from your current index. 4. Once `reindex-from-snapshot` is complete, captured traffic is replayed from AWS MSK to the target cluster by the traffic replayer. 5. Performance and behavior of traffic sent to the source and target clusters are compared by reviewing logs and metrics. 6. After confirming the target cluster’s functionality meets expectations, clients are redirected to the new target. From 187da006d051163ae6a1d3a1f1d0b37dbc04ceb1 Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Mon, 25 Nov 2024 12:08:37 -0600 Subject: [PATCH 10/14] Apply suggestions from code review Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- _migrations/index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/_migrations/index.md b/_migrations/index.md index b4eed54074..8311302fcd 100644 --- a/_migrations/index.md +++ b/_migrations/index.md @@ -47,7 +47,7 @@ The Metadata migration tool integrated into the Migration CLI can be used indepe ### reindex-from-snapshot -`reindex-from-snapshot` reindexes data from an existing snapshot on Elastic Container Service (ECS) workers that coordinate the migration of documents from an existing snapshot, reindexing the documents in parallel to a target cluster. +`Reindex-from-Snapshot` (RFS) reindexes data from an existing snapshot. Workers on Elastic Container Services (ECS) coordinate the migration of documents from an existing snapshot, reindexing the documents in parallel to a target cluster. ### Target cluster @@ -55,7 +55,7 @@ The destination cluster for migration or comparison in an A/B test. ### Architecture overview -The Migration assistant architecture is based on the use of an AWS cloud infrastructure, but most tools are designed to be cloud-independent. A local containerized version of this solution is also available. +The Migration assistant architecture is based on the use of an AWS Cloud infrastructure, but most tools are designed to be cloud-independent. A local containerized version of this solution is also available. The design deployed in AWS is as follows: From b30ad5712fca2666cb2b56d2392f6d07ca715573 Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Tue, 26 Nov 2024 09:16:23 -0600 Subject: [PATCH 11/14] Apply suggestions from code review Co-authored-by: Nathan Bower Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- _migrations/index.md | 16 ++++++++-------- .../is-migration-assistant-right-for-you.md | 18 +++++++++--------- 2 files changed, 17 insertions(+), 17 deletions(-) diff --git a/_migrations/index.md b/_migrations/index.md index 8311302fcd..a46604e613 100644 --- a/_migrations/index.md +++ b/_migrations/index.md @@ -9,7 +9,7 @@ has_toc: false # Migration Assistant for OpenSearch -Migrations Assistant for OpenSearch aids you in successfully performing an end-to-end, zero-downtime migration to OpenSearch from other search providers. It helps with the following scenarios: +Migration Assistant for OpenSearch aids you in successfully performing an end-to-end, zero-downtime migration to OpenSearch from other search providers. It helps with the following scenarios: - **Metadata migration**: Migrating cluster metadata, such as index settings, aliases, and templates. - **Backfill migration**: Migrating existing or historical data from a source to a target cluster. @@ -21,7 +21,7 @@ This user guide focuses on conducting a comprehensive migration involving both e It's crucial to note that migration strategies are not universally applicable. This guide provides a detailed methodology, based on certain assumptions detailed throughout, emphasizing the importance of robust engineering practices to ensure a successful migration. {: .tip } -## Key components of Migration Assistant +## Key components The following are the key components of Migration Assistant. @@ -35,7 +35,7 @@ A console that provides a migration-specific CLI and offers a variety of tools t ### Traffic capture proxy -This component is designed for HTTP RESTful traffic. It forwards traffic to the source cluster and also splits and channels this traffic to a stream-processing service for later playback. +This component is designed for HTTP RESTful traffic. It forwards traffic to the source cluster and also splits and channels this traffic to a stream processing service for later playback. ### Traffic Replayer @@ -47,7 +47,7 @@ The Metadata migration tool integrated into the Migration CLI can be used indepe ### reindex-from-snapshot -`Reindex-from-Snapshot` (RFS) reindexes data from an existing snapshot. Workers on Elastic Container Services (ECS) coordinate the migration of documents from an existing snapshot, reindexing the documents in parallel to a target cluster. +`Reindex-from-Snapshot` (RFS) reindexes data from an existing snapshot. Workers on Amazon Elastic Container Service (Amazon ECS) coordinate the migration of documents from an existing snapshot, reindexing the documents in parallel to a target cluster. ### Target cluster @@ -55,16 +55,16 @@ The destination cluster for migration or comparison in an A/B test. ### Architecture overview -The Migration assistant architecture is based on the use of an AWS Cloud infrastructure, but most tools are designed to be cloud-independent. A local containerized version of this solution is also available. +The Migration Assistant architecture is based on the use of an AWS Cloud infrastructure, but most tools are designed to be cloud independent. A local containerized version of this solution is also available. The design deployed in AWS is as follows: ![Migration architecture overview]({{site.url}}{{site.baseurl}}/images/migrations/migration-architecture-overview.svg) 1. Client traffic is directed to the existing cluster. -2. An Application Load Balancer (ALB) with capture proxies relays traffic to a source while replicating data to Amazon Managed Streaming for Apace Kafka (AWS MSK). +2. An Application Load Balancer with capture proxies relays traffic to a source while replicating data to Amazon Managed Streaming for Apache Kafka (Amazon MSK). 3. Using the migration console, you can initiate metadata migration to establish indexes, templates, component templates, and aliases on the target cluster. 4. With continuous traffic capture in place, you can use a `reindex-from-snapshot` process to capture data from your current index. -4. Once `reindex-from-snapshot` is complete, captured traffic is replayed from AWS MSK to the target cluster by the traffic replayer. +4. Once `reindex-from-snapshot` is complete, captured traffic is replayed from Amazon MSK to the target cluster by the traffic replayer. 5. Performance and behavior of traffic sent to the source and target clusters are compared by reviewing logs and metrics. -6. After confirming the target cluster’s functionality meets expectations, clients are redirected to the new target. +6. After confirming that the target cluster's functionality meets expectations, clients are redirected to the new target. diff --git a/_migrations/is-migration-assistant-right-for-you.md b/_migrations/is-migration-assistant-right-for-you.md index a5a0fce958..1719645ff2 100644 --- a/_migrations/is-migration-assistant-right-for-you.md +++ b/_migrations/is-migration-assistant-right-for-you.md @@ -10,7 +10,7 @@ Before deciding if Migration Assistant is right for you, it's important to asses Migration Assistant addresses gaps found in other migration solutions, but in some cases, alternative tools may be a better fit. -For instance, if you need to upgrade more than one major version, such as moving from Elasticsearch 6.8 to OpenSearch 2.15, Migration Assistant allows you to do this in a single hop. In contrast, other options like rolling upgrades or snapshot restore would require multiple steps, as they cannot handle major version jumps without reindexing your data. Additionally, if you need to capture live traffic and perform a zero-downtime migration, Migration Assistant would be the right choice. +For instance, if you need to upgrade more than one major version, such as moving from Elasticsearch 6.8 to OpenSearch 2.15, Migration Assistant allows you to do this in a single hop. In contrast, other options like rolling upgrades or snapshot restore would require multiple steps because they cannot handle major version jumps without reindexing your data. Additionally, if you need to capture live traffic and perform a zero-downtime migration, Migration Assistant would be the right choice. There are also tools available for migrating cluster configuration, templates, and aliases, which can be helpful depending on the complexity of your setup. These tools streamline the migration process by preserving critical settings and custom configurations. @@ -37,7 +37,7 @@ The tooling is designed to work with other cloud provider platforms, but it is n ### Future migration paths -To see the roadmap for OpenSearch migrations, go to [OpenSarch Migration - Roadmap](https://github.com/orgs/opensearch-project/projects/229/views/1). +To see the OpenSearch migrations roadmap, go to [OpenSearch Migrations - Roadmap](https://github.com/orgs/opensearch-project/projects/229/views/1). ## Supported components @@ -47,13 +47,13 @@ Before starting a migration, consider the scope of the components involved. The | :--- |:--- | :--- | | **Documents** | Yes | Migrate existing data with Reindex from Snapshot (RFS) and live traffic with Capture and Replay. | | **Index settings** | Yes | Migrate with the metadata migration tool. | -| **Index mappings** | Yes | Migrate with metadata migration tool. | -| **Index templates** | Yes | Migrate with metadata migration tool. | -| **Component templates** | Yes | Migrate with Metadata Migration Tool | -| **Aliases** | Yes | Migrate with metadata migration tool | -| **ISM policies** | Expected in 2025 | Manually migrate using API | -| **Elasticsearch Kibana dashboards** | Expected in 2025 | This tool is only needed when used to migrate Elasticsearch Kibana Dashboards to OpenSearch Dashboards. To start, export JSON files from Kibana and import into OpenSearch Dashboards; before importing, use the [`dashboardsSanitizer`](https://github.com/opensearch-project/opensearch-migrations/tree/main/dashboardsSanitizer) tool on X-Pack visualizations like Canvas and Lens in Kibana Dashboards, as they may require recreation for compatibility with OpenSearch. | -| **Security constructs** | No | Configure roles and permissions based on cloud provider recommendations. For example, if using AWS, leverage IAM for enhanced security management. | +| **Index mappings** | Yes | Migrate with the metadata migration tool. | +| **Index templates** | Yes | Migrate with the metadata migration tool. | +| **Component templates** | Yes | Migrate with the metadata migration tool. | +| **Aliases** | Yes | Migrate with the metadata migration tool. | +| **Index State Management (ISM) policies** | Expected in 2025 | Manually migrate using an API. | +| **Elasticsearch Kibana dashboards** | Expected in 2025 | This tool is only needed when used to migrate Elasticsearch Kibana Dashboards to OpenSearch Dashboards. To start, export JSON files from Kibana and import them into OpenSearch Dashboards; before importing, use the [`dashboardsSanitizer`](https://github.com/opensearch-project/opensearch-migrations/tree/main/dashboardsSanitizer) tool on X-Pack visualizations like Canvas and Lens in Kibana Dashboards, as they may require recreation for compatibility with OpenSearch. | +| **Security constructs** | No | Configure roles and permissions based on cloud provider recommendations. For example, if using AWS, leverage AWS Identity and Access Management (IAM) for enhanced security management. | | **Plugins** | No | Check plugin compatibility; some Elasticsearch plugins may not have direct equivalents in OpenSearch. | From f9aa933ea72a4566cd12a2003e1bd8e6148a1a36 Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Tue, 26 Nov 2024 09:17:09 -0600 Subject: [PATCH 12/14] Update is-migration-assistant-right-for-you.md Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- _migrations/is-migration-assistant-right-for-you.md | 6 ------ 1 file changed, 6 deletions(-) diff --git a/_migrations/is-migration-assistant-right-for-you.md b/_migrations/is-migration-assistant-right-for-you.md index 1719645ff2..e905229d96 100644 --- a/_migrations/is-migration-assistant-right-for-you.md +++ b/_migrations/is-migration-assistant-right-for-you.md @@ -55,9 +55,3 @@ Before starting a migration, consider the scope of the components involved. The | **Elasticsearch Kibana dashboards** | Expected in 2025 | This tool is only needed when used to migrate Elasticsearch Kibana Dashboards to OpenSearch Dashboards. To start, export JSON files from Kibana and import them into OpenSearch Dashboards; before importing, use the [`dashboardsSanitizer`](https://github.com/opensearch-project/opensearch-migrations/tree/main/dashboardsSanitizer) tool on X-Pack visualizations like Canvas and Lens in Kibana Dashboards, as they may require recreation for compatibility with OpenSearch. | | **Security constructs** | No | Configure roles and permissions based on cloud provider recommendations. For example, if using AWS, leverage AWS Identity and Access Management (IAM) for enhanced security management. | | **Plugins** | No | Check plugin compatibility; some Elasticsearch plugins may not have direct equivalents in OpenSearch. | - - -## Migration assessment questionnaire - -If you're having trouble deciding whether migration -[OpenSearch Migrations Assessment - Version 5.docx](https://github.com/user-attachments/files/17177220/OpenSearch.Migrations.Assessment.-.Version.5.docx). \ No newline at end of file From 74316b24359e39c2b0b6a8f2829f2f7fa76d2953 Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Tue, 26 Nov 2024 09:18:41 -0600 Subject: [PATCH 13/14] Apply suggestions from code review Co-authored-by: Nathan Bower Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- _migrations/index.md | 2 +- _migrations/is-migration-assistant-right-for-you.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/_migrations/index.md b/_migrations/index.md index a46604e613..e3b2657e1a 100644 --- a/_migrations/index.md +++ b/_migrations/index.md @@ -13,7 +13,7 @@ Migration Assistant for OpenSearch aids you in successfully performing an end-to - **Metadata migration**: Migrating cluster metadata, such as index settings, aliases, and templates. - **Backfill migration**: Migrating existing or historical data from a source to a target cluster. -- **Live traffic migration**: Replicating live ongoing traffic from source to target cluster. +- **Live traffic migration**: Replicating live ongoing traffic from a source to a target cluster. - **Comparative tooling**: Comparing the performance and behaviors of an existing cluster with a prospective new one. This user guide focuses on conducting a comprehensive migration involving both existing and live data with zero downtime and the option to back out of a migration. diff --git a/_migrations/is-migration-assistant-right-for-you.md b/_migrations/is-migration-assistant-right-for-you.md index e905229d96..c00d49f80d 100644 --- a/_migrations/is-migration-assistant-right-for-you.md +++ b/_migrations/is-migration-assistant-right-for-you.md @@ -45,7 +45,7 @@ Before starting a migration, consider the scope of the components involved. The | Component | Supported | Recommendations | | :--- |:--- | :--- | -| **Documents** | Yes | Migrate existing data with Reindex from Snapshot (RFS) and live traffic with Capture and Replay. | +| **Documents** | Yes | Migrate existing data with `Reindex-from-Snapshot` (RFS) and live traffic with capture and replay. | | **Index settings** | Yes | Migrate with the metadata migration tool. | | **Index mappings** | Yes | Migrate with the metadata migration tool. | | **Index templates** | Yes | Migrate with the metadata migration tool. | From 6a9e212632d413360d820e96313c8e68364d009a Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Tue, 26 Nov 2024 14:07:47 -0600 Subject: [PATCH 14/14] Update _migrations/is-migration-assistant-right-for-you.md Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- _migrations/is-migration-assistant-right-for-you.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_migrations/is-migration-assistant-right-for-you.md b/_migrations/is-migration-assistant-right-for-you.md index c00d49f80d..6a09e44206 100644 --- a/_migrations/is-migration-assistant-right-for-you.md +++ b/_migrations/is-migration-assistant-right-for-you.md @@ -45,7 +45,7 @@ Before starting a migration, consider the scope of the components involved. The | Component | Supported | Recommendations | | :--- |:--- | :--- | -| **Documents** | Yes | Migrate existing data with `Reindex-from-Snapshot` (RFS) and live traffic with capture and replay. | +| **Documents** | Yes | Migrate existing data with `reindex-from-snapshot` (RFS) and live traffic with Capture and Replay. | | **Index settings** | Yes | Migrate with the metadata migration tool. | | **Index mappings** | Yes | Migrate with the metadata migration tool. | | **Index templates** | Yes | Migrate with the metadata migration tool. |