Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: OpenSearch Serverless docs - V1.5.0 release #323

Merged
merged 1 commit into from
Apr 25, 2023
Merged

docs: OpenSearch Serverless docs - V1.5.0 release #323

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion pages/architecture.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -249,10 +249,16 @@ through security groups inbound rules.
data.all uses Amazon OpenSearch to index datasets information
for optimal search experience on the catalog.

By default, Amazon OpenSearch Service cluster is created, however users have the ability to use Amazon OpenSearch
Serverless collection instead by enabling a corresponding feature flag in `cdk.json`.

- Amazon OpenSearch cluster is running inside a VPC and private
subnets.
- If using Amazon OpenSearch Serverless collection, it is only accessible through OpenSearch
Serverless–managed VPC endpoints.
- It is accessible only by data.all resources like Lambda
functions and ECS tasks thanks to enforced security groups inbound rules.
functions and ECS tasks thanks to enforced security groups inbound rules in case of OpenSearch cluster, or access
policies in case of Amazon OpenSearch Serverless.
- It is encrypted at rest with AWS KMS customer managed key (CMK).

### AWS Lambda OpenSearch Handler
Expand Down
3 changes: 2 additions & 1 deletion pages/code.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,8 @@ In the pipeline stack `PipelineStack` we deploy the following, which deploy the
- `DBMigrationStack`: tool to migrate between Aurora versions of the database
- `LambdaApiStack` : Lambda Function stack
- `MonitoringStack` : CloudWatch alarms and monitoring resources
- `OpenSearchStack`: OpenSearch cluster - data.all central catalog
- `OpenSearchStack`: OpenSearch cluster - data.all central catalog (default)
- `OpenSearchServerlessStack`: OpenSearch Serverless collection - data.all central catalog (if enabled)
- `ParamStoreStack` : AWS SSM parameters
- `S3ResourcesStack` : S3 resources
- `SecretsManagerStack` : AWS SSM Secrets
Expand Down
59 changes: 34 additions & 25 deletions pages/deploy/deploy_aws.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,8 @@ of our repository. Open it, you should be seen something like:
"prod_sizing": "boolean_SET_INFRA_SIZING_TO_PROD_VALUES_IF_TRUE|DEFAULT=true",
"enable_cw_rum": "boolean_SET_CLOUDWATCH_RUM_APP_MONITOR|DEFAULT=false",
"enable_cw_canaries": "boolean_SET_CLOUDWATCH_CANARIES_FOR_FRONTEND_TESTING|DEFAULT=false",
"shared_dashboards_sessions": "string_TYPE_SESSION_SHARED_DASHBOARDS|(reader, anonymous) DEFAULT=anonymous"
"shared_dashboards_sessions": "string_TYPE_SESSION_SHARED_DASHBOARDS|(reader, anonymous) DEFAULT=anonymous",
"enable_opensearch_serverless": "boolean_USE_OPENSEARCH_SERVERLESS|DEFAULT=false"
}
]
}
Expand All @@ -122,30 +123,31 @@ have listed and defined all the parameters of the cdk.json file. If you still ha
and find 2 examples of cdk.json files.


| **General Parameters** | **Optional/Required** | **Definition** |
|----------------------------------------|-----------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| tooling_vpc_id | Optional | The VPC ID for the tooling account. If not provided, **a new VPC** will be created. |
| tooling_region | Optional | The AWS region for the tooling account where the AWS CodePipeline pipeline will be created. (default: eu-west-1) |
| git_branch | Optional | The git branch name can be leveraged to deploy multiple AWS CodePipeline pipelines to the same tooling account. (default: main) |
| git_release | Optional | If set to **true**, CI/CD pipeline RELEASE stage is enabled. This stage releases a version out of the current branch. (default: false) |
| quality_gate | Optional | If set to **true**, CI/CD pipeline quality gate stage is enabled. (default: true) |
| resource_prefix | Optional | The prefix used for AWS created resources. It must be in lower case without any special character. (default: dataall) |
| **Deployment environments Parameters** | **Optional/Required** | **Definition** |
| ---------------------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| envname | REQUIRED | The name of the deployment environment (e.g dev, qa, prod,...). It must be in lower case without any special character. |
| account | REQUIRED | The AWS deployment account (deployment account N) |
| region | REQUIRED | The AWS deployment region |
| with_approval | Optional | If set to **true** an additional step on AWS CodePipeline to require user approval before proceeding with the deployment. (default: false) |
| vpc_id | Optional | The VPC ID for the deployment account. If not provided, **a new VPC** will be created. |
| vpc_endpoints_sg | Optional | The VPC endpoints security groups to be use by AWS services to connect to VPC endpoints. If not assigned, NAT outbound rule is used. |
| internet_facing | Optional | If set to **true** CloudFront is used for hosting data.all UI and Docs and APIs are public. If false, ECS is used to host static sites and APIs are private. (default: true) |
| **General Parameters** | **Optional/Required** | **Definition** |
|----------------------------------------|-----------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| tooling_vpc_id | Optional | The VPC ID for the tooling account. If not provided, **a new VPC** will be created. |
| tooling_region | Optional | The AWS region for the tooling account where the AWS CodePipeline pipeline will be created. (default: eu-west-1) |
| git_branch | Optional | The git branch name can be leveraged to deploy multiple AWS CodePipeline pipelines to the same tooling account. (default: main) |
| git_release | Optional | If set to **true**, CI/CD pipeline RELEASE stage is enabled. This stage releases a version out of the current branch. (default: false) |
| quality_gate | Optional | If set to **true**, CI/CD pipeline quality gate stage is enabled. (default: true) |
| resource_prefix | Optional | The prefix used for AWS created resources. It must be in lower case without any special character. (default: dataall) |
| **Deployment environments Parameters** | **Optional/Required** | **Definition** |
| ---------------------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| envname | REQUIRED | The name of the deployment environment (e.g dev, qa, prod,...). It must be in lower case without any special character. |
| account | REQUIRED | The AWS deployment account (deployment account N) |
| region | REQUIRED | The AWS deployment region |
| with_approval | Optional | If set to **true** an additional step on AWS CodePipeline to require user approval before proceeding with the deployment. (default: false) |
| vpc_id | Optional | The VPC ID for the deployment account. If not provided, **a new VPC** will be created. |
| vpc_endpoints_sg | Optional | The VPC endpoints security groups to be use by AWS services to connect to VPC endpoints. If not assigned, NAT outbound rule is used. |
| internet_facing | Optional | If set to **true** CloudFront is used for hosting data.all UI and Docs and APIs are public. If false, ECS is used to host static sites and APIs are private. (default: true) |
| custom_domain | Optional* | Custom domain configuration: hosted_zone_name, hosted_zone_id, and certificate_arn. If internet_facing parameter is **false** then custom_domain is REQUIRED for ECS ALB integration with ACM and HTTPS. It is optional when internet_facing is true. |
| ip_ranges | Optional | Used only when internet_facing parameter is **false** to allow API Gateway resource policy to allow these IP ranges in addition to the VPC's CIDR block. |
| apig_vpce | Optional | Used only when internet_facing parameter is **false**. If provided, it will be used for API Gateway otherwise a new VPCE will be created. |
| prod_sizing | Optional | If set to **true**, infrastructure sizing is adapted to prod environments. Check additional resources section for more details. (default: true) |
| enable_cw_rum | Optional | If set to **true** CloudWatch RUM monitor is created to monitor the user interface (default: false) |
| enable_cw_canaries | Optional | If set to **true**, CloudWatch Synthetics Canaries are created to monitor the GUI workflow of principle features (default: false) |
| shared_dashboard_sessions | Optional | Either 'anonymous' or 'reader'. It indicates the type of Quicksight session used for Shared Dashboards (default: 'anonymous') |
| ip_ranges | Optional | Used only when internet_facing parameter is **false** to allow API Gateway resource policy to allow these IP ranges in addition to the VPC's CIDR block. |
| apig_vpce | Optional | Used only when internet_facing parameter is **false**. If provided, it will be used for API Gateway otherwise a new VPCE will be created. |
| prod_sizing | Optional | If set to **true**, infrastructure sizing is adapted to prod environments. Check additional resources section for more details. (default: true) |
| enable_cw_rum | Optional | If set to **true** CloudWatch RUM monitor is created to monitor the user interface (default: false) |
| enable_cw_canaries | Optional | If set to **true**, CloudWatch Synthetics Canaries are created to monitor the GUI workflow of principle features (default: false) |
| shared_dashboard_sessions | Optional | Either 'anonymous' or 'reader'. It indicates the type of Quicksight session used for Shared Dashboards (default: 'anonymous') |
| enable_opensearch_serverless | Optional | If set to **true** Amazon OpenSearch Serverless collection is created and used instead of Amazon OpenSearch Service domain (default: false) |

**Example 1**: Basic deployment: this is an example of a minimum configured cdk.json file.

Expand Down Expand Up @@ -385,4 +387,11 @@ their default behavior explained in the [documentation](https://docs.aws.amazon.

### Troubleshooting - The CodePipeline Pipeline fails with CodeBuild Error Code "AccountLimitExceededException"
Sometimes, we run into the following error *"Error calling startBuild: Cannot have more than 1 builds in queue for the account"*.
Nothing is wrong with the code itself, CodeBuild quotas have been hit. Just click on **Retry**.
Nothing is wrong with the code itself, CodeBuild quotas have been hit. Just click on **Retry**.

### I would like to migrate to Amazon OpenSearch Serverless
If you have deployed data.all with Amazon OpenSearch and would like to migrate to Amazon OpenSearch Serverless,
you would need to migrate the indexes to your new cluster. Although data.all currently does not provide an automated
migration tool, it is possible to do so manually using the following approaches:
- [Migrate your indexes to Amazon OpenSearch Serverless with Logstash](https://aws.amazon.com/blogs/big-data/migrate-your-indexes-to-amazon-opensearch-serverless-with-logstash/).
- [Migrating Amazon OpenSearch Service indexes using remote reindex](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/remote-reindex.html)