Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deploy release branches to staging ECS environment #4758

Merged
merged 9 commits into from
Nov 5, 2018
76 changes: 30 additions & 46 deletions .circleci/bin/ecs-deploy.sh
Original file line number Diff line number Diff line change
@@ -1,64 +1,48 @@
#!/bin/bash

sudo apt-get -y install python3-pip wget jq
sudo apt-get -y install python3-pip wget
sudo pip3 install awscli
wget https://github.com/mikefarah/yq/releases/download/2.0.1/yq_linux_amd64 -O /tmp/yq
sudo mv /tmp/yq /usr/local/bin/yq
sudo chmod +x /usr/local/bin/yq

if [ -z "${AWS_REGION}" ]; then
AWS_REGION=us-west-2
fi

ENVIRONMENT=staging
SERVICE_DIR_NAME=.reaction/devops/aws/services
SERVICES=$(ls ${SERVICE_DIR_NAME})
CLUSTER=core
export core_CIRCLE_SHA1=$CIRCLE_SHA1

for SERVICE in $SERVICES; do
DISABLED=$(echo $SERVICE | grep disabled)
if [ "${DISABLED}" == "${SERVICE}" ]; then
continue
fi
echo "START PROCESSING SERVICE ${SERVICE}"

cd ${SERVICE_DIR_NAME}/${SERVICE}

PROPEL_CONFIG_FILE="propel-${ENVIRONMENT}.yaml"
if [ ! -f ${PROPEL_CONFIG_FILE} ]; then
echo "Propel configuration file not found!"
exit 1
fi
PROPEL_CONFIG_FILE="propel.yaml"
if [ ! -f ${PROPEL_CONFIG_FILE} ]; then
echo "Propel configuration file not found!"
exit 1
fi

ENV_NAME_UPPERCASE=$(echo $ENVIRONMENT | awk '{print toupper($0)}')
AWS_ACCESS_KEY_ID_VAR_NAME=CLOUDFORMATION_${ENV_NAME_UPPERCASE}_AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY_VAR_NAME=CLOUDFORMATION_${ENV_NAME_UPPERCASE}_AWS_SECRET_ACCESS_KEY
ENV_NAME_UPPERCASE=$(echo $ENVIRONMENT | awk '{print toupper($0)}')
AWS_ACCESS_KEY_ID_VAR_NAME=CLOUDFORMATION_${ENV_NAME_UPPERCASE}_AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY_VAR_NAME=CLOUDFORMATION_${ENV_NAME_UPPERCASE}_AWS_SECRET_ACCESS_KEY

if [ "${!AWS_ACCESS_KEY_ID_VAR_NAME}" ]; then
AWS_ACCESS_KEY_ID=${!AWS_ACCESS_KEY_ID_VAR_NAME}
fi
if [ "${!AWS_ACCESS_KEY_ID_VAR_NAME}" ]; then
AWS_ACCESS_KEY_ID=${!AWS_ACCESS_KEY_ID_VAR_NAME}
fi

if [ "${!AWS_SECRET_ACCESS_KEY_VAR_NAME}" ]; then
AWS_SECRET_ACCESS_KEY=${!AWS_SECRET_ACCESS_KEY_VAR_NAME}
fi
if [ "${!AWS_SECRET_ACCESS_KEY_VAR_NAME}" ]; then
AWS_SECRET_ACCESS_KEY=${!AWS_SECRET_ACCESS_KEY_VAR_NAME}
fi

mkdir -p ~/.aws
echo "[default]" > ~/.aws/credentials
echo "aws_access_key_id = ${AWS_ACCESS_KEY_ID}" >> ~/.aws/credentials
echo "aws_secret_access_key = ${AWS_SECRET_ACCESS_KEY}" >> ~/.aws/credentials
mkdir -p ~/.aws
echo "[default]" > ~/.aws/credentials
echo "aws_access_key_id = ${AWS_ACCESS_KEY_ID}" >> ~/.aws/credentials
echo "aws_secret_access_key = ${AWS_SECRET_ACCESS_KEY}" >> ~/.aws/credentials
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@griggheo Just curious, why write these to a file when AWS lib typically already looks for env variables with these names? Seems more secure to avoid any credentials in files.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It didn't work for me when I specified the AWS keys without actually writing them to the credentials file. I'll try again. Agreed it's better not to save them to disk.


echo "[default]" > ~/.aws/config
echo "region = ${AWS_REGION}" >> ~/.aws/config
echo "[default]" > ~/.aws/config
echo "region = ${AWS_REGION}" >> ~/.aws/config

echo Running aws s3 cp s3://${S3_PROPEL_ARTIFACTS_BUCKET}/propel-linux-amd64 ./propel
aws s3 cp s3://${S3_PROPEL_ARTIFACTS_BUCKET}/propel-linux-amd64 ./propel
echo Running aws s3 cp s3://${S3_PROPEL_ARTIFACTS_BUCKET}/propel-linux-amd64 ./propel
aws s3 cp s3://${S3_PROPEL_ARTIFACTS_BUCKET}/propel-linux-amd64 ./propel

sudo mv propel /usr/local/bin/propel
sudo chmod +x /usr/local/bin/propel
sudo mv propel /usr/local/bin/propel
sudo chmod +x /usr/local/bin/propel

RELEASE_DESCRIPTION="CircleCI build URL: ${CIRCLE_BUILD_URL}"
propel release create --deploy --descr "${RELEASE_DESCRIPTION}" -f ${PROPEL_CONFIG_FILE}

echo "END PROCESSING SERVICE ${SERVICE}"

cd -
done
RELEASE_DESCRIPTION="CircleCI build URL: ${CIRCLE_BUILD_URL}"
echo storefront_CIRCLE_SHA1=$storefront_CIRCLE_SHA1
propel release create --deploy --descr "${RELEASE_DESCRIPTION}" -f ${PROPEL_CONFIG_FILE} --env $ENVIRONMENT --cluster $CLUSTER
6 changes: 3 additions & 3 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -339,9 +339,9 @@ workflows:
- deploy-to-ecs:
requires:
- docker-push
# filters:
# branches:
# only: /^release-2\.0.\d+$/
filters:
branches:
only: /^release*/
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Were we going to do /^release-2*/ instead? There is currently a release-1.17.0 branch and could at some point be release-1.17.1. Should branches like that overwrite staging deployment?

cc @spencern

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aldeed @griggheo Good catch, I don't think we should overwrite the staging branch with 1.x releases. I think we can probably get by without having a staging branch for 1.x as we're putting minimal development effort towards it

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK I'll change it to /^release-2.*/ for now.

- deploy-docs:
requires:
- test-unit
Expand Down
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
.fileStorage/
.vscode
.idea
.env
.env*
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's an open issue to add a .env file for this repo for local development, and that will involve creating a file named .env.example which will need to be committed (same pattern as in other repos). Can you change this ignore rule so that .env.example will be committed when we add that?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will do.

.c9
*.csv
*.dat
Expand Down
33 changes: 0 additions & 33 deletions .reaction/devops/aws/services/service-reaction/propel-staging.yaml

This file was deleted.

6 changes: 0 additions & 6 deletions .reaction/devops/aws/services/service-starterkit/.env

This file was deleted.

This file was deleted.

33 changes: 33 additions & 0 deletions propel.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
services:
- name: reaction-core
certificate_arn: arn:aws:acm:us-west-2:773713188930:certificate/c2979a7a-7b84-43ed-b3a7-24b0256a1b9f
root_domain: staging.reactioncommerce.com
dns_name: reaction-core
desired_task_count: 2
min_task_count: 0
max_task_count: 2
min_healthy_percent: 100
max_percent: 200
alb_listener_port: 80
alb_listener_path: /
alb_health_check_path: /health
task-definition:
name: staging-core
containers:
- name: core
cpu: 500
memory: 1800
port_mappings:
- container_port: 3000
host_port: 3000
image: reactioncommerce/reaction
image_tag: release-2.0.0-rc.5
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this will need to be updated every time we make a new release branch?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The image_tag is used only if you deploy using propel from the command line. CircleCI deployments will automatically use the SHA1 of the newly built Docker image.

env_params:
- name: REACTION_AUTH
- name: REACTION_EMAIL
- name: REACTION_USER
- name: MONGO_URL
- name: ROOT_URL
- name: HYDRA_ADMIN_URL
- name: HYDRA_OAUTH2_INTROSPECT_URL
- name: SKIP_FIXTURES
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this list need to change when we add env variables? Where do the values come from? Maybe we can make this more straightforward when we do #4706

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Those are there for informational purposes only. I can remove them once we have the .env.example file.