Improved Self-Managed Enterprise scaling & troubleshooting #38863
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,135 @@ | ||||||
| --- | ||||||
| products: oss-enterprise | ||||||
| --- | ||||||
|
|
||||||
| # Scaling Airbyte After Installation | ||||||
|
|
||||||
| Once you've completed the initial installation of Airbyte Self-Managed Enterprise, the next crucial step is scaling your setup as needed to ensure optimal performance and reliability as your data integration needs grow. This guide will walk you through best practices and strategies for scaling Airbyte in an enterprise environment. | ||||||
|
|
||||||
| ## Concurrent Syncs | ||||||
|
|
||||||
| The primary indicator of increased resource usage in Airbyte is the number of concurrent syncs running at any given time. Each concurrent sync requires at least 3 additional connector pods to be running at once (`orchestrator`, `read`, `write`). This means that 10 concurrent syncs requires 30 additional pods in your namespace. Connector pods last only for the duration of a sync, and will be appended by the ID of the ongoing job. | ||||||
|
|
||||||
| If your deployment of Airbyte is intended to run many concurrent syncs at once (e.g. an overnight backfill), we recommend provisioning an increased number of instances. Some connectors are memory and CPU intensive, while others are not. Using an infrastructure monitoring tool, we recommend measuring the following at all times: | ||||||
| * Requested CPU % | ||||||
| * CPU Usage % | ||||||
| * Requested Memory % | ||||||
| * Memory Usage % | ||||||
|
|
||||||
| With high CPU or Memory usage, we recommend scaling up your Airbyte deployment to a higher number of nodes, or reducing the maximum resource usage by any given connector pod. If high _requested_ CPU or memory usage is blocking new pods from being scheduled, while _used_ CPU or memory is low, you may modify connector pod provisioning defaults in your `values.yml` file: | ||||||
|
|
||||||
| ```yaml | ||||||
| global: | ||||||
| edition: "enterprise" | ||||||
| ... | ||||||
| jobs: | ||||||
| resources: | ||||||
| limits: | ||||||
| cpu: | ||||||
| memory: | ||||||
| requests: | ||||||
| cpu: | ||||||
| memory: | ||||||
| ``` | ||||||
|
|
||||||
| If your Airbyte deployment is underprovisioned, you may often notice occasional 'stuck jobs' that remain in-progress for long periods, with eventual failures related to unavailable pods. If you begin to see such errors, we recommend you follow the troubleshooting steps above. | ||||||
|
There was a problem hiding this comment.
Suggested change
|
||||||
|
|
||||||
| ## Multiple Node Groups | ||||||
|
|
||||||
| To reduce the blast radius of an underprovisioned Airbyte deployment, we recommend placing 'static' workloads (`webapp`, `server`, etc.) on one Kubernetes node group, while placing job-related workloads (connector pods) on a different Kubernetes node group. This ensures that UI or API availability is unlikely to be impacted by the number of concurrent syncs. | ||||||
|
|
||||||
| <details> | ||||||
| <summary>Configure Airbyte Self-Managed Enterprise to run in two node groups</summary> | ||||||
|
|
||||||
| ```yaml | ||||||
| airbyte-bootloader: | ||||||
| nodeSelector: | ||||||
| type: static | ||||||
|
|
||||||
| server: | ||||||
| nodeSelector: | ||||||
| type: static | ||||||
|
|
||||||
| keycloak: | ||||||
| nodeSelector: | ||||||
| type: static | ||||||
|
|
||||||
| keycloak-setup: | ||||||
| nodeSelector: | ||||||
| type: static | ||||||
|
|
||||||
| temporal: | ||||||
| nodeSelector: | ||||||
| type: static | ||||||
|
|
||||||
| webapp: | ||||||
| nodeSelector: | ||||||
| type: static | ||||||
|
|
||||||
| worker: | ||||||
| nodeSelector: | ||||||
| type: jobs | ||||||
|
|
||||||
| workload-launcher: | ||||||
| nodeSelector: | ||||||
| type: static | ||||||
| ## Pods spun up by the workload launcher will run in the 'jobs' node group. | ||||||
| extraEnvs: | ||||||
| - name: JOB_KUBE_NODE_SELECTORS | ||||||
| value: type=jobs | ||||||
| - name: SPEC_JOB_KUBE_NODE_SELECTORS | ||||||
| value: type=jobs | ||||||
| - name: CHECK_JOB_KUBE_NODE_SELECTORS | ||||||
| value: type=jobs | ||||||
| - name: DISCOVER_JOB_KUBE_NODE_SELECTORS | ||||||
| value: type=jobs | ||||||
|
|
||||||
| orchestrator: | ||||||
| nodeSelector: | ||||||
| type: jobs | ||||||
|
|
||||||
| workload-api-server: | ||||||
| nodeSelector: | ||||||
| type: jobs | ||||||
| ``` | ||||||
|
|
||||||
| </details> | ||||||
|
|
||||||
| ## High Availability | ||||||
|
|
||||||
| You may wish to implement high availability (HA) to minimize downtime and ensure continuous data integration processes. Please note that this requires provisioning Airbyte on a larger number of Nodes, which may increase your licensing fees. For a typical HA deployment, you will want a VPC with subnets in at least two (and preferably three) availability zones (AZs). | ||||||
|
|
||||||
| We particularly recommend having multiple instances of `worker` and `server` pods: | ||||||
|
|
||||||
| ```yaml | ||||||
| worker: | ||||||
| replicaCount: 2 | ||||||
|
|
||||||
| server: | ||||||
| replicaCount: 2 | ||||||
| ``` | ||||||
|
|
||||||
| Furthermore, you may want to implement a primary-replica setup for the database (e.g., PostgreSQL) used by Airbyte. The primary database handles write operations, while replicas handle read operations, ensuring data availability even if the primary fails. | ||||||
|
|
||||||
| ## Disaster Recovery (DR) Regions | ||||||
|
|
||||||
| For business-critical applications of Airbyte, you may want to configure a Disaster Recovery (DR) cluster for Airbyte. We do not support assisting customers with DR deployments at this time. However, we offer a few high level suggestions: | ||||||
| 1. We strongly recommend configuring an external database, external log storage and external connector secret management. | ||||||
| 2. We strongly recommend that your DR cluster is also an instance of Self-Managed Enterprise, kept at the same version as your prod instance. | ||||||
|
|
||||||
| ## DEBUG Logs | ||||||
|
|
||||||
| We recommend turning off `DEBUG` logs for any non-testing use of Self-Managed Airbyte. Failing to do while running at-scale syncs may result in the `server` pod being overloaded, preventing most of the deployment for operating as normal. | ||||||
|
There was a problem hiding this comment.
Suggested change
|
||||||
|
|
||||||
| ## Schema Discovery Timeouts | ||||||
|
|
||||||
| While configuring a database source connector with hundreds to thousands of tables, each with many columns, the one-time `discover` mechanism - by which we discover the topology of your source - may run for a long time and exceed Airbyte's timeout duration. Should this be the case, you may increase Airbyte's timeout limit as follows: | ||||||
|
There was a problem hiding this comment.
Suggested change
There was a problem hiding this comment. It could be useful to link to the Airbyte protocol definition for "Discover" as well: |
||||||
|
|
||||||
| ```yaml | ||||||
| server: | ||||||
| extraEnvs: | ||||||
| - name: HTTP_IDLE_TIMEOUT | ||||||
| value: 20m | ||||||
| - name: READ_TIMEOUT | ||||||
| value: 30m | ||||||
| ``` | ||||||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would it make sense to expand on the first sentence and hint at more specific requirements listed later? Something like "Airbyte Self-Managed Enterprise must be deployed on a Kubernetes cluster managed with Helm. This is to enable Airbyte's best performance and scale. We support enterprise deployments on AWS, GCP, or Azure using services outlined in this document."