Skip to content

Commit

Permalink
Deployment docs and sidebar cleanup (#20965)
Browse files Browse the repository at this point in the history
  • Loading branch information
Amruta-Ranade authored Jan 3, 2023
1 parent 15689ad commit cae6396
Show file tree
Hide file tree
Showing 21 changed files with 209 additions and 181 deletions.
15 changes: 6 additions & 9 deletions docs/api-documentation.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,24 +2,21 @@

## Open-Source API

:::caution

For Airbyte Open-Source you don't need the API Token for Authentication!
All endpoints are possible to access using the API without it.
Airbyte Open Source doesn't require an API Token for Authentication.
All endpoints are acccessible without the token.

:::
:::caution

Our Open-Source API (configuration API) is still in an alpha state and might change. You won’t lose any functionality, but you may need to update your code to catch up to any backwards incompatible changes in the API.
:::

**Check out the** [**Open-Source API documentation**](https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html)**.**

Don't hesitate to ask us on [Slack](https://slack.airbyte.io), if you have any questions about it.
Contact us on [Slack](https://slack.airbyte.io) if you have any questions about it.

---

## Airbyte Cloud API

:::info

Airbyte Cloud will also support API access in 2023. See [Airbyte roadmap](https://app.harvestr.io/roadmap/view/pQU6gdCyc/airbyte-roadmap) for more details.
Airbyte Cloud will support API access in 2023. See [Airbyte roadmap](https://app.harvestr.io/roadmap/view/pQU6gdCyc/airbyte-roadmap) for more details.

54 changes: 27 additions & 27 deletions docs/deploying-airbyte/on-aws-ec2.md
Original file line number Diff line number Diff line change
@@ -1,35 +1,34 @@
# Deploy Airbyte on AWS (Amazon EC2)

This page guides you through deploying Airbyte Open Source on an Amazon EC2 instance by setting up the deployment environment, installing and starting Airbyte, and connecting it to the Amazon EC2 instance.
This page guides you through deploying Airbyte Open Source on an [Amazon EC2 instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts.html) by setting up the deployment environment, installing and starting Airbyte, and connecting it to the Amazon EC2 instance.

> INFO
>
> The instructions have been tested on Amazon Linux 2 AMI (HVM)
:::info

The instructions have been tested on Amazon Linux 2 AMI (HVM).

:::

## Requirements

- To test Airbyte, we recommend a `t2.medium` instance
- To deploy Airbyte in a production environment, we recommend a `t2.large` instance
- Make sure your Docker Desktop app is running to ensure all services run smoothly

[Create and download an SSH key to connect to the instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/create-key-pairs.html)
- [Create and download an SSH key to connect to the instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/create-key-pairs.html)

## Set up the environment

1. Create an Amazon EC2 instance.

To connect to your instance, run the following command on your local terminal:
1. To connect to your instance, run the following command on your local terminal:

```
``` bash
SSH_KEY=~/Downloads/dataline-key-airbyte.pem # the file path you downloaded the key
INSTANCE_IP=REPLACE_WITH_YOUR_INSTANCE_IP # you can find your IP address in the EC2 console under the Instances tab
INSTANCE_IP=REPLACE_WITH_YOUR_INSTANCE_IP # find your IP address in the EC2 console under the Instances tab
chmod 400 $SSH_KEY # or ssh will complain that the key has the wrong permissions
ssh -i $SSH_KEY ec2-user@$INSTANCE_IP # connect to the aws ec2 instance AMI and the your private IP address
```

2. To install Docker, run the following command in your SSH session on the instance terminal:

```
``` bash
sudo yum update -y
sudo yum install -y docker
sudo service docker start
Expand All @@ -38,50 +37,51 @@ sudo usermod -a -G docker $USER

3. To install `docker-compose`, run the following command in your ssh session on the instance terminal:

```
``` bash
sudo wget https://github.com/docker/compose/releases/download/1.26.2/docker-compose-$(uname -s)-$(uname -m) -O /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
docker-compose --version
```

4. To close the SSH connection, run the following command in your SSH session on the instance terminal:

```
``` bash
logout
```

## Install and start Airbyte

In your local terminal, run the following commands:

1. Connect to your instance
1. Connect to your instance:

```
``` bash
ssh -i $SSH_KEY ec2-user@$INSTANCE_IP
```

2. Install Airbyte
2. Install Airbyte:

```
``` bash
mkdir airbyte && cd airbyte
wget https://raw.githubusercontent.com/airbytehq/airbyte/master/{.env,docker-compose.yaml}
docker-compose up -d # run the Docker container
```

## Connect to Airbyte

> DANGER
>
> We strongly recommend not exposing Airbyte on Internet available ports for security reasons.
:::caution

1. Create an SSH tunnel for port 8000
For security reasons, we strongly recommend not exposing Airbyte on Internet available ports.

> INFO
>
> If you want to use different ports, modify `API_URL` in your .env file and restart Airbyte.
> Run the following commands in your workstation terminal from the downloaded key folder:
:::

```
1. Create an SSH tunnel for port 8000:

If you want to use different ports, modify `API_URL` in your .env file and restart Airbyte.
Run the following commands in your workstation terminal from the downloaded key folder:


``` bash
# In your workstation terminal
SSH_KEY=~/Downloads/dataline-key-airbyte.pem
ssh -i $SSH_KEY -L 8000:localhost:8000 -N -f ec2-user@$INSTANCE_IP
Expand Down
13 changes: 4 additions & 9 deletions docs/deploying-airbyte/on-azure-vm-cloud-shell.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Deploy Airbyte on Azure (Cloud Shell)

This page guides you through deploying Airbyte Open Source on a Microsoft Azure VM by setting up the deployment environment, installing and starting Airbyte, and connecting it to the VM.
This page guides you through deploying Airbyte Open Source on a [Microsoft Azure VM](https://learn.microsoft.com/en-us/azure/virtual-machines/) by setting up the deployment environment, installing and starting Airbyte, and connecting it to the VM.

:::info

Expand Down Expand Up @@ -81,13 +81,11 @@ Download Airbyte and deploy it in the VM using Docker Compose:
wget https://raw.githubusercontent.com/airbytehq/airbyte/master/{.env,docker-compose.yaml}
```

4. Start Airbyte by running the following command:
4. To start Airbyte, run the following command:

```bash
sudo docker-compose up -d
```
This step takes about two minutes to complete. When done, you will see the cursor prompt.


## Connect to Airbyte

Expand All @@ -100,14 +98,11 @@ Test a remote connection to your VM locally and verify that Airbyte is up and ru
```bash
ssh -N -L 8000:localhost:8000 -i <your SSH key file> <admin username>@<IP address>
```
As a result, nothing happens. The cursor prompt keeps blinking.

4. Open a web browser and navigate to `http://localhost:8000`. You will see Airbyte’s landing page.

:::danger

For security reasons, it is strongly recommended not to expose Airbyte on Internet available ports. Future versions will add support for SSL and authentication.

:::caution
For security reasons, we strongly recommend not exposing Airbyte on Internet available ports.
:::

## Troubleshooting
Expand Down
6 changes: 3 additions & 3 deletions docs/deploying-airbyte/on-digitalocean-droplet.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
# Deploy Airbyte on DigitalOcean

This page guides you through deploying Airbyte Open Source on a DigitalOcean droplet by setting up the deployment environment, and installing and starting Airbyte.
This page guides you through deploying Airbyte Open Source on a [DigitalOcean droplet](https://docs.digitalocean.com/products/droplets/how-to/create/) by setting up the deployment environment, and installing and starting Airbyte.

Alternatively, you can deploy Airbyte on DigitalOcean in one click using their [marketplace](https://cloud.digitalocean.com/droplets/new?onboarding_origin=marketplace&appId=95451155&image=airbyte&utm_source=deploying-airbyte_on-digitalocean-droplet).

## Requirements

- To test Airbyte, we recommend a $20/month droplet.
- To deploy Airbyte in a production environment, we recommend a $40/month instance.
- To test Airbyte, we recommend a $20/month droplet
- To deploy Airbyte in a production environment, we recommend a $40/month instance

## Set up the Environment

Expand Down
9 changes: 3 additions & 6 deletions docs/deploying-airbyte/on-gcp-compute-engine.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,3 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';

# Deploy Airbyte on GCP (Compute Engine)

This page guides you through deploying Airbyte Open Source on a [Google Cloud Platform (GCP) Compute Engine instance](https://cloud.google.com/compute/docs/instances) by setting up the deployment environment, installing and starting Airbyte, and connecting it to the GCP instance.
Expand All @@ -13,8 +10,8 @@ The instructions have been tested on a `Debian GNU/Linux 10` VM instance.

## Requirements

- To test Airbyte, we recommend an `e2.medium` instance and provision at least 30GBs of disk per node.
- To deploy Airbyte in a production environment, we recommend a `n1-standard-2` instance.
- To test Airbyte, we recommend an `e2.medium` instance and provision at least 30GBs of disk per node
- To deploy Airbyte in a production environment, we recommend a `n1-standard-2` instance

## Set up the environment

Expand Down Expand Up @@ -95,7 +92,7 @@ docker-compose up -d
## Connect to Airbyte

:::caution
Warning: For security reasons, we strongly recommended not exposing Airbyte publicly.
For security reasons, we strongly recommended not exposing Airbyte publicly.
:::

1. In your local terminal, create an SSH tunnel to connect the GCP instance to Airbyte:
Expand Down
59 changes: 33 additions & 26 deletions docs/deploying-airbyte/on-kubernetes-via-helm.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,8 @@ Airbyte allows scaling sync workloads horizontally using Kubernetes. The core co

## Quickstart

If you don't want to configure your own Kubernetes cluster and Airbyte instance, you can use the free, open-source project [Plural](https://www.plural.sh/) to bring up a Kubernetes cluster and Airbyte for you. Use [this guide](on-plural.md) to get started.<br/>
If you don't want to configure your own Kubernetes cluster and Airbyte instance, you can use the free, open-source project [Plural](https://www.plural.sh/) to bring up a Kubernetes cluster and Airbyte for you. Use [this guide](on-plural.md) to get started.

Alternatively, you can deploy Airbyte on [Restack](https://www.restack.io) to provision your Kubernetes cluster on AWS. Follow [this guide](on-restack.md) to get started.

## Getting Started
Expand Down Expand Up @@ -34,33 +35,39 @@ If you do not already have the CLI tool `kubectl` installed, please follow [thes

Configure `kubectl` to connect to your cluster by using `kubectl use-context my-cluster-name`.

* For GKE
* Configure `gcloud` with `gcloud auth login`.
* On the Google Cloud Console, the cluster page will have a `Connect` button, which will give a command to run locally that looks like
For GKE:

1. Configure `gcloud` with `gcloud auth login`.
2. On the Google Cloud Console, the cluster page will have a `Connect` button, which will give a command to run locally that looks like

`gcloud container clusters get-credentials $CLUSTER_NAME --zone $ZONE_NAME --project $PROJECT_NAME`.

* Use `kubectl config get-contexts` to show the contexts available.
* Run `kubectl config use-context $GKE_CONTEXT` to access the cluster from `kubectl`.
* For EKS
* [Configure your AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html) to connect to your project.
* Install [eksctl](https://eksctl.io/introduction/)
* Run `eksctl utils write-kubeconfig --cluster=<CLUSTER NAME>` to make the context available to `kubectl`
* Use `kubectl config get-contexts` to show the contexts available.
* Run `kubectl config use-context <eks context>` to access the cluster with `kubectl`.
3. Use `kubectl config get-contexts` to show the contexts available.
4. Run `kubectl config use-context $GKE_CONTEXT` to access the cluster from `kubectl`.

For EKS:

1. [Configure your AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html) to connect to your project.
2. Install [eksctl](https://eksctl.io/introduction/)
3. Run `eksctl utils write-kubeconfig --cluster=<CLUSTER NAME>` to make the context available to `kubectl`
4. Use `kubectl config get-contexts` to show the contexts available.
5. Run `kubectl config use-context <eks context>` to access the cluster with `kubectl`.

### Install helm

To install helm simply run:

* For MacOS
* `brew install helm`
* For Linux
* Download installer script `curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3`
* Assign required premissions `chmod 700 get_helm.sh`
* Run script `./get_helm.sh`
For MacOS:

`brew install helm`

### Adding Helm Repository
For Linux:

1. Download installer script `curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3`
2. Assign required premissions `chmod 700 get_helm.sh`
3. Run script `./get_helm.sh`

### Add Helm Repository

From now charts are stored in helm-repo thus there're no need to clone the repo each time you need to deploy the chart.

Expand All @@ -85,7 +92,8 @@ airbyte-oss/webapp 0.30.23 0.39.37-alpha Helm chart to de
airbyte-oss/worker 0.30.23 0.39.37-alpha Helm chart to deploy airbyte-worker
```

## Deploying airbyte
## Deploy Airbyte

### Default deployment

If you don't intend to customise your deployment, you can deploy airbyte as is with default values.
Expand All @@ -95,20 +103,19 @@ In order to do so, run the command:
helm install %release_name% airbyte/airbyte
```

It'll deploy airbyte with default configuration
### Custom deployment

### Configuring deployment
In order to customize your deployment, you need to create `values.yaml` file in the local folder and populate it with default configuration override values.

In order to configure deployment, you need to create values.yaml file in local folder and populate it with default configuration override values.
`values.yaml` example can be located in [charts/airbyte](https://github.com/airbytehq/airbyte/blob/master/charts/airbyte/values.yaml) folder of the Airbyte repository.

values.yaml example can be located in [charts/airbyte](https://github.com/airbytehq/airbyte/blob/master/charts/airbyte/values.yaml) folder of the airbyte repository.
After specifying your own configuration, run the following command:

After specifying your own configuration, proceed with chart deployment by running
```text
helm install --values path/to/values.yaml %release_name% airbyte/airbyte
```

## Migration from old chart to new ones
## Migrate from old charts to new ones

Starting from `0.39.37-alpha` we've revisited helm charts structure and separated all components of airbyte into their own independent charts, thus by allowing our developers to test single component without deploying airbyte as a whole and by upgrading single component at a time.

Expand Down
Loading

0 comments on commit cae6396

Please sign in to comment.