Skip to content

Latest commit

 

History

History
125 lines (87 loc) · 3.18 KB

on-aws-ec2.md

File metadata and controls

125 lines (87 loc) · 3.18 KB
Raw

On AWS (EC2)

{% hint style="info" %} The instructions have been tested on Amazon Linux 2 AMI (HVM) {% endhint %}

Create a new instance

  • Launch a new instance

  • Select instance AMI

  • Select instance type
    • For testing out Airbyte, a t2.medium instance is likely sufficient.
    • For long-running Airbyte installations, we recommend a t2.large instance.

  • Next: Configure Instance Details
    • You can tune parameters or keep the defaults
  • Next: Add Storage
    • You can tune parameters or keep the defaults
  • Next: Add Tags
    • You can tune parameters or keep the defaults
  • Next: Configure Security Groups
    • We are going to allow network for ssh

  • Review and Launch
  • Launch
  • Create a ssh key so you can connect to the instance
    • Download the key (and don't lose it or you won't be able to connect to the instance)

  • Launch Instances

  • Wait for the instance to become Running

Install environment

{% hint style="info" %} This part assumes that you have access to a terminal on your workstation {% endhint %}

  • Connect to your instance
# In your workstation terminal
SSH_KEY=~/Downloads/airbyte-key.pem # or wherever you've downloaded the key
INSTANCE_IP=REPLACE_WITH_YOUR_INSTANCE_IP
chmod 400 $SSH_KEY # or ssh will complain that the key has the wrong permissions
ssh -i $SSH_KEY ec2-user@$INSTANCE_IP
  • Install docker
# In your ssh session on the instance terminal
sudo yum update -y
sudo yum install -y docker
sudo service docker start
sudo usermod -a -G docker $USER
  • Install docker-compose
# In your ssh session on the instance terminal
sudo wget https://github.com/docker/compose/releases/download/1.26.2/docker-compose-$(uname -s)-$(uname -m) -O /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
docker-compose --version
  • Close the ssh connection to ensure the group modification is taken into account
# In your ssh session on the instance terminal
logout

Install & start Airbyte

  • Connect to your instance
# In your workstation terminal
ssh -i $SSH_KEY ec2-user@$INSTANCE_IP
  • Install Airbyte
# In your ssh session on the instance terminal
mkdir airbyte && cd airbyte
wget https://raw.githubusercontent.com/airbytehq/airbyte/master/{.env,docker-compose.yaml}
docker-compose up -d

Connect to Airbyte

{% hint style="danger" %} For security reasons, we strongly recommend to not expose Airbyte on Internet available ports. Future versions will add support for SSL & Authentication. {% endhint %}

  • Create ssh tunnel.
# In your workstation terminal
ssh -i $SSH_KEY -L 8000:localhost:8000 -L 8001:localhost:8001 -N -f ec2-user@$INSTANCE_IP

Troubleshooting

If you encounter any issues, just connect to our Slack. Our community will help!