Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use CF7 for rolling deploys and restarts #1150

Merged
merged 4 commits into from
Aug 26, 2020
Merged

Use CF7 for rolling deploys and restarts #1150

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
c63dda8
Use CF7 for rolling deploys and restarts
Jkrzy Aug 19, 2020
27aba98
Update docs to reflect new deploy approach
Jkrzy Aug 26, 2020
6f51f34
Merge branch 'main' into latest-clouds
Jkrzy Aug 26, 2020
4598319
Merge branch 'main' into latest-clouds
Jkrzy Aug 26, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 37 additions & 23 deletions .circleci/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,14 @@
#
# Check https://circleci.com/docs/2.0/language-python/ for more details
#
cf-docker-image: &CF_DOCKER_IMAGE
docker:
- image: 18fgsa/cloud-foundry-cli
environment:
- TZ=America/New_York
- CF_API: https://api.fr.cloud.gov
install-cf7: &install-cf7
run:
name: Install CF7
command: |
curl -L -o cf7.deb 'https://packages.cloudfoundry.org/stable?release=debian64&version=v7&source=github'
sudo dpkg -i cf7.deb
rm cf7.deb
cf7 api https://api.fr.cloud.gov

version: 2
jobs:
Expand Down Expand Up @@ -84,50 +86,62 @@ jobs:
- ./*

deploy_to_staging:
<<: *CF_DOCKER_IMAGE

docker:
- image: cimg/base:2020.01
environment:
- TZ=America/New_York
- CF_SPACE: staging
- CF_APP: tock
- CF_MANIFEST: manifest-staging.yml
steps:
- attach_workspace:
at: .
- *install-cf7
- run:
name: Login to cloud.gov Staging
command: cf login -a ${CF_API} -u ${CF_DEPLOYER_USERNAME_STAGING} -p ${CF_DEPLOYER_PASSWORD_STAGING}
name: Login to cloud.gov
command: cf7 login -u ${CF_DEPLOYER_USERNAME_STAGING} -p {CF_DEPLOYER_PASSWORD_STAGING} -o gsa-18f-tock -s ${CF_SPACE}
- run:
name: Save version to file system
command: echo ${CIRCLE_SHA1} > tock/VERSION
- run:
name: deploy Tock Staging to cloud.gov
command: cf_deploy.sh tock gsa-18f-tock staging manifest-staging.yml
name: Deploying Tock Staging to cloud.gov
command: cf7 push $CF_APP --strategy rolling -f ${CF_MANIFEST}

deploy_to_production:
<<: *CF_DOCKER_IMAGE

docker:
- image: cimg/base:2020.01
environment:
- TZ=America/New_York
- CF_SPACE: prod
- CF_MANIFEST: manifest-production.yml
steps:
- attach_workspace:
at: .
- *install-cf7
- run:
name: Login to cloud.gov Production
command: cf login -a ${CF_API} -u ${CF_DEPLOYER_USERNAME_PRODUCTION} -p ${CF_DEPLOYER_PASSWORD_PRODUCTION}
command: cf7 login -u ${CF_DEPLOYER_USERNAME_PRODUCTION} -p ${CF_DEPLOYER_PASSWORD_PRODUCTION} -o gsa-18f-tock -s ${CF_SPACE}
- run:
name: Save version to file system
command: echo ${CIRCLE_TAG} > tock/VERSION
- run:
name: Deploy Tock Production to cloud.gov
command: cf_deploy.sh tock gsa-18f-tock prod manifest-production.yml
command: cf7 push tock --strategy rolling -f ${CF_MANIFEST}

recycle_production:
<<: *CF_DOCKER_IMAGE

docker:
- image: cimg/base:2020.01
environment:
- TZ=America/New_York
- CF_SPACE: prod
steps:
- *install-cf7
- run:
name: Login to cloud.gov Production
command: cf login -a ${CF_API} -u ${CF_DEPLOYER_USERNAME_PRODUCTION} -p ${CF_DEPLOYER_PASSWORD_PRODUCTION}
- run:
name: Install cf-rolling-restart
command: cf install-plugin -f -r CF-Community "cf-rolling-restart"
command: cf7 login -u ${CF_DEPLOYER_USERNAME_PRODUCTION} -p ${CF_DEPLOYER_PASSWORD_PRODUCTION} -o gsa-18f-tock -s ${CF_SPACE}
- run:
name: Performing a rolling restart of Tock Production instances
command: cf rolling-restart tock
command: cf restart tock --strategy rolling

workflows:
version: 2
Expand Down
68 changes: 21 additions & 47 deletions docs/deployment-process.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,7 @@ Download the Cloud Foundry CLI according to the [cloud.gov instructions][].

[cloud.gov instructions]: https://docs.cloud.gov/getting-started/setup/

You will also need to install the [`autopilot`](https://github.com/contraband/autopilot)
plugin for Cloud Foundry, which is used for zero-downtime deploys:

```shell
# Install the plugin
cf install-plugin autopilot -f -r CF-Community
```
We use the V7 Cloud Foundry CLI. If you're upgrading from V6, checkout [the CLI docs for instructions](https://github.com/cloudfoundry/cli).

Tock will be deployed to the GovCloud instance of cloud.gov:

Expand All @@ -23,18 +17,19 @@ Tock will be deployed to the GovCloud instance of cloud.gov:
cf login -a api.fr.cloud.gov --sso
```

After this deployment, you'll need to target the org and space you want to work with. For example, if you wanted to work with the dev space:
`cf target -o gsa-18f-tock -s dev`
After authenticating, you'll need to target the org and space you want to work with. For example, if you wanted to work with the dev space:

```
cf target -o gsa-18f-tock -s dev
```

Manifest files, which contain import deploy configuration settings, are located
in the root directory of this project, prefixed with `manifest-` and ending in a
`.yml` file extension.

During local development and continuous integration testing,
`pipenv install --dev` is used. This installs both development
and production dependencies. During deployments, the Cloud Foundry
Python buildpack generates a `requirements.txt` file with `pipenv lock -r`
and installs only production dependencies.
and production dependencies. During deployments, the `requirements.txt` file is used to install dependencies.

### Cloud Foundry structure

Expand Down Expand Up @@ -92,9 +87,8 @@ cf cups tock-credentials -p credentials-<ENVIRONMENT>.json
# Binding the service to the app
cf bind-service <APP_INSTANCE> tock-credentials

# Restaging the app to make use of the updated credentials. This will cause
# downtime in the application. It is better to use zero-downtime-push instead.
cf restage <APP_INSTANCE>
# Restaging the app to make use of the updated credentials.
cf restage <APP_INSTANCE> --strategy rolling
```

You can update the user-provided service with the following commands:
Expand All @@ -103,9 +97,8 @@ You can update the user-provided service with the following commands:
# Uploading the new credentials to the service
cf uups tock-credentials -p credentials-staging.json

# Restaging the app to make use of the updated credentials. This will cause
# downtime in the application. It is better to use zero-downtime-push instead.
cf restage <APP_INSTANCE>
# Restaging the app to make use of the updated credentials.
cf restage <APP_INSTANCE> --strategy rolling
```

#### Database service
Expand Down Expand Up @@ -152,7 +145,7 @@ Should you need to, you can push directly to tock.app.cloud.gov with the followi

```sh
cf target -o gsa-18f-tock -s staging
cf zero-downtime-push tock-staging -f manifest-staging.yml
cf push tock-staging -f manifest-staging.yml --strategy rolling
```

### Production servers
Expand All @@ -162,20 +155,14 @@ be made against the `main` branch following the [_Automated Releases to
Production_](#automated-releases-to-production) workflow.

In some cases, you may need to make a manual deployment to production. If this is the case, please make
sure you're using the Cloud Foundry [autopilot plugin](https://github.com/contraband/autopilot).
sure you're using the rolling deployment strategy with `--strategy rolling`

To deploy, first make sure you're targeting the prod space:

```sh
cf target -o gsa-18f-tock -s prod
```

If you don't already have the autopilot plugin, you can install it by running the following:

```sh
cf install-plugin autopilot -f -r CF-Community
```

Create a `VERSION` file with the name of the version that is being deployed to
production either with the Git SHA1 for the latest commit or the Git tag for the
latest release:
Expand All @@ -188,33 +175,20 @@ echo $(git rev-parse HEAD | head -c 7) > tock/VERSION
echo $(git describe --abbrev=0 --tags) > tock/VERSION
```

Then use the autopilot plugin's `zero-downtime-push` command to deploy:
Then use the CLI to deploy:

```sh
cf zero-downtime-push tock -f manifest-production.yml
cf push tock -f manifest-production.yml --strategy rolling
```

#### Troubleshooting failed manual zero-downtime deployments
#### Troubleshooting failed deployments

If at any point the deployment fails, there should still be zero-downtime for
the production instance. Please verify that the Tock applications that are
running are named correctly and cleaned up. In the following example commands,
the variables should be replaced with the values found in the previous
commands.
If at any point the deployment fails, we can use the CLI to recover to our previous state.

- `${STOPPED_TOCK_APP}` — The application that reads `stopped` from `cf apps`.
- `${TOCK_VERNERABLE_APP_NAME}` — The application that reads `-venerable` from
`cf apps`.
https://docs.cloudfoundry.org/devguide/deploy-apps/rolling-deploy.html#cancel

```sh
# List all applications in the targeted space
cf apps

# Delete the stopped application
cf delete ${STOPPED_TOCK_APP} -f

# Rename the started application to match project conventions
cf rename ${TOCK_VERNERABLE_APP_NAME} tock
```
cf cancel-deployment tock
```

### Logs
Expand Down Expand Up @@ -310,7 +284,7 @@ number after the period (e.g., `v20180131.1` turns into `v20180131.2`).

Once you push this tag up to GitHub, draft or assign it to an already
drafted release in GitHub. CircleCI will deploy this tag to the Production
instance of Tock using CF Autopilot.
instance of Tock.


## Maintenance Mode
Expand Down