This project contains Terraform scripts that are used to generate a container hosting environment in AWS. This environment is in the eu-west-2 region.
This environment allows the deployment of externally accessible Application containers and private API containers. Example build pipelines for an application, api and NPM module are also provided.
The available scripts are divided into three main areas: security, infrastructure and build pipelines. When creating a new environment scripts will need to be initially executed in that order. There is an additional area, ssm_config which is optional depending upon the type of installation and can be used to configure the parameter store prior to running th build pipelines is so desired.
All of the scripts will require terraform init to be executed in the correct directory prior to running any other terraform commands to ensure modules and plugins are brought up to date/= or more accurately in line with the terraform code.
[TOC]
Some additional documentation is also available:
/terraform/security
The security scripts generate a number of IAM user groups and policies documents. They do not create any actual users.
The following groups are created:
- CCS_System_Administration
- CCS_User_Administration
- CCS_Infrastructure_Administration
- CCS_Application_Developer
- CCS_API_Developer
- CCS_Code_Build_Pipeline
- CCS_Cognito_Administration
- CCS_Terraform_Execution
- CCS_Developer_API_Access
- CCS_MFA_ManageOwnDevice
Only a very small number of users should be a member of the system administration group. To have complete system administration privileges a user will need to be a member of the system administration, infrastructure administration and code build pipeline groups.
The CCS_Terraform_Execution group is available to give additional AWS users the capability to run Terraform scripts via the shared-state.
Note that members of the user administration group are not able to add users to the system administration group.
More information is available in the terraform/security directory.
/terraform/infrastructure
The infrastructure scripts generate a complete AWS environment for deploying containers. The AWS access keys used when executing the scripts must correspond to an AWS user with permission to actually create all of the required AWS assets. Making the user a member of the CCS_System_Administration, CCS_Infrastructure_Administration and CCS_Code_Build_Pipeline IAM groups will ensure this.
- VPC across three availability zone: Only Zone 'a' is populated at present.
- Public, private and management subnets within each availability zone.
- Private DNS zone in Route 53 assigned to the VPC.
- Network ACL rules.
- Security groups to control access to various features and functions.
- A basic bastion host for management.
- Two ECS clusters:
- CCSDEV_app_cluster for 'Applications' in the public subnet.
- CCSDEV_api_cluster for 'APIs' in the private subnet.
- Public application load balancer for the application cluster.
- HTTP and HTTPS using AWS managed certificates.
- Internal application load balancer for the api cluster.
- HTTP and HTTPS using AWS managed certificates.
- Example RDS Postgres database server.
- Example Redis based caching
- Example Elastic Search domain.
- CloudWatch dashboard definition example
- Cognito user pool
NOTE In this release the example database and Elastic Search instances can be accessed from the Application and API clusters. The intention is that this is restricted to the API cluster in the next release.
A file with the suffix .auto.tfvars must be created that defines the IP addresses from which access is allowed and the root domain name defined in Route53. An example file, config.auto.tfvars.example is provided:
# The domain name that the components will sit beneath
domain_name = "example.com"
# The subdomain beneath the main domain name that will be used for
# internal (api) components - this defaults to "internal"
domain_internal_prefix = "internal"
# Whether HTTPS should be available across the load balancers
# along with appropriate certificates
enable_https = false
# A map of external CIDR blocks that should have SSH access
"ssh_access_cidrs" = {
"office" = "192.0.2.0/24"
}
# A map of external CIDR blocks that should have HTTP(S) access
"app_access_cidrs" = {
"office" = "192.0.2.0/24"
"guests" = "198.51.100.0/24"
}
When the infrastructure scripts are executed a number of settings are written to the EC2 Parameter Store as secure strings. These settings are then used when creating the build pipelines to generate environment variables via the ECS Task Definition which is used to create the container. In this way database and Elastic Search connection details are passed to the Applications and APIs.
/terraform/build
There are examples that create AWS CodePipeline and CodeBuild configurations for an application (app1, app2), and an api (api1, api2). These will take source code from GitHub and, ultimately, deploy an updated container image to the correct ECS cluster. There is also an example that publishes an updated NPM module to https://www.npmjs.com.
Each of these examples will generate AWS CodeBuild and CodePipeline configurations. The content of these will vary with the type of build. For example, an NPM module build will not create or update any ECS task definitions.
All of these builds require access to a GitHub account for a GitHub token. This must be entered into the EC2 parameter store as a secure string called: ccs-build_github_token. They each use pre-defined buildspec files that are located within the terraform build module as <prefix>_buildspec.yml.
/terraform/build/app1
This uses the contents of the CMpExampleApp1 repository. It contains a minimal NodeJS/Express application and a Dockerfile for defining the container image.
The file main.tf defines the attributes of the build and identifies the location of the source code in GitHub, the name of the cluster to which it will be deployed and the type of build to be performed. It also contains a variable that specifies the root domain name for the Route 53 zone to which an 'A' record will be added to make the application accessible. The name and prefix settings will be combined to form the container image name within ECR.
When executed the scripts will define the build pipeline and this will trigger the process of building and deploying the example application. Eventually, assuming the build succeeds, the application will be accessible as http://app1.<domain>.
/terraform/build/app2
This uses the contents of the CMpExampleApp2 repository. It contains a minimal Ruby on Rails application and a Dockerfile for defining the container image.
The file main.tf defines the attributes of the build and identifies the location of the source code in GitHub, the name of the cluster to which it will be deployed and the type of build to be performed. It also contains a variable that specifies the root domain name for the Route 53 zone to which an 'A' record will be added to make the application accessible. The name and prefix settings will be combined to form the container image name within ECR.
When executed the scripts will define the build pipeline and this will trigger the process of building and deploying the example application. Eventually, assuming the build succeeds, the application will be accessible as http://app2.<domain>.
/terraform/build/api1
This uses the contents of the CMpExampleApi1 repository. It contains a minimal Java/Spring Boot REST API and a Dockerfile for defining the container image.
The file main.tf defines the attributes of the build and identifies the location of the source code in GitHub, the name of cluster to which it will be deployed and the type of build to be performed. It also contains a variable that specifies the root domain name for the Route 53 zone to which an 'A' record will be added to make the api accessible. The default value for this is the private zone only usable within the VPC. The name and prefix settings will be combined to form the container image name within ECR.
When executed the scripts will define the build pipeline and this will trigger the process of building and deploying the example api. Eventually, assuming the build succeeds, the api will be accessible as http://api1.<domain>. Note that with the settings in the example this is deployed to the api cluster within a private subnet. To test, one option is to ssh to the basiton host and execute curl. See the documentation within the CMpExampleApi1 repository for details of the REST interface.
/terraform/build/api2
This uses the contents of the CMpExampleApi2 repository. It contains a minimal Python + Flask REST API and a Dockerfile for defining the container image.
The file main.tf defines the attributes of the build and identifies the location of the source code in GitHub, the name of cluster to which it will be deployed and the type of build to be performed. It also contains a variable that specifies the root domain name for the Route 53 zone to which an 'A' record will be added to make the api accessible. The default value for this is the private zone only usable within the VPC. The name and prefix settings will be combined to form the container image name within ECR.
When executed the scripts will define the build pipeline and this will trigger the process of building and deploying the example api. Eventually, assuming the build succeeds, the api will be accessible as http://api2.<domain>. Note that with the settings in the example this is deployed to the api cluster within a private subnet. To test, one option is to ssh to the bastion host and execute curl. See the documentation within the CMpExampleApi2 repository for details of the REST interface.
/terraform/build/npm1
This uses the contents of the CCSExampleNPMModule repository. It contains a minimal JavaScript module that can be built and published as an NPM module.
The file main.tf defines the attributes of the build and identifies the location of the source code in GitHub. The example uses a bespoke build type of npm-publish which will install the package, run any tests and then attempt to publish the package to the npmjs registry - In order for this to be successful, there is a predefined npm auth token stored within the AWS parameter store (ccs-build_npm_token).
When executed the scripts will define the build pipeline and this will trigger the process of building and publishing the example NPM module.
/terraform/build/image-ruby
This uses the contents of the CMpDevBuildImage_Ruby repository which contains a Dockerfile and associated context that can be built and published as docker image within the AWS Container Registry for use in other builds.
The file main.tf defines the attributes of the build and identifies the location of the source code in GitHub. The image build type uses a 2-stage pipeline that will build the container image and then publish it to the AWS Container Registry.
The produced image can be used in subsequent component builds by setting the build_type variable to 'custom' and the 'build_image' variable to '{Image Prefix}/{Image Name}' as specified when the image was built. For example:
build_image = "ccs/ruby"
/terraform/build/cmp-maintenance
This uses the contents of the crown-marketplace-maintenance to build and deploy a small nodeJS application that displays a static HTML page. The file main.tf contains a setting, catch_all that will configure the application load balancer to direct any request to a non-existent application to this page.
NOTE : The operation of the catch-all settings is dependent on the ordering of rules within the application load balancer listener rules. It is not always possible for Terraform to maintain this list in the correct order if any manual changes are made to the routing rules.
/terraform/build/crown-marketplace-legacy
This is the actual Marketplace application taken from the crown-marketplace-legacy repository. It is a Ruby application and the build pipeline will use the custom Ruby build image described previously. This must be available as an image in the AWS Elastic Container Repository (ECR) before the application pipeline is created.
This is configured to use just hostname based routing and because of this registers the DNS entry for the hostname/load balancer.
/terraform/build/crown-marketplace
This is the actual Marketplace application taken from the crown-marketplace repository. It is a Ruby application and the build pipeline will use the custom Ruby build image described previously. This must be available as an image in the AWS Elastic Container Repository (ECR) before the application pipeline is created.
This is configured to use hostname and path based routing and does not register the DNS entry for the hostname/load balancer. As such it relies on the Crown Marketplace Legacy Application being deployed to function.
/terraform/build/crown-marketplace-upload
This is the actual Marketplace application taken from the crown-marketplace repository. However it is deployed to the API Cluster and, as can be seen in the file main.tf, sets the APP_HAS_UPLOAD_PRIVILEGES environment variable. This allows it to be used to upload data to the marketplace application. Because it is deployed to the API Cluster this capability can only be accessed from within the VPC.
/terraform/build/upload-supply-teacher-data
This pipeline uses the crown-marketplace-data repository to detect changes to the data available for the application and, when changes are committed, will upload the supply teacher related data to the application by POSTing the data to the Marketplace application deployed to the API Cluster. The data is uploaded using the same process described in the crown-marketplace repository documentation.
When the build pipelines are created AWS Simple Notification Service (SNS) topics are also created. Each pipeline will have two topics, that represent successful and failed builds. AWS console users can subscribe to these topics to receive, via email, notification of the corresponding event.
Topic: [pipeline name]-success
Notification generated when the pipeline state changes to SUCCEEDED.
Topic: [pipeline name]-failure
Notification generated when the pipeline state changes to FAILED.
AWS Console users who are members of the CCS_Application_Developer and CCS_Application_Developer IAM groups receive permissions that allow them to subscribe to the topics.
The build pipeline scripts will ensure that a number of environment variables are passed to the running containers.
Of particular importance are those variables that allow an app container to determine the URL for invoking an api. These are:
- CCS_API_PROTOCOL
- CCS_API_BASE_URL
- CCS_APP_PROTOCOL
- CCS_APP_BASE_URL
The PROTOCOL variables will contain either http or https. The BASE_URL variables contain the domain name for accessing applications or apis.
An application can construct a URL for accessing api1 using:
CCS_API_PROTOCOL + "://" + "api1." + CCS_API_BASE_URL
The example NPM module actually contains a simple class for this purpose and is used by the example application.
Database, Redis and Elastic Search connection information is also supplied as environment variables:
- CCS_DEFAULT_DB_URL
- CCS_DEFAULT_DB_TYPE
- CCS_DEFAULT_DB_HOST
- CCS_DEFAULT_DB_PORT
- CCS_DEFAULT_DB_NAME
- CCS_DEFAULT_DB_USER
- CCS_DEFAULT_DB_PASSWORD
- CCS_REDIS_HOST
- CCS_REDIS_PORT
- CCS_DEFAULT_ES_ENDPOINT
CCS_DEFAULT_DB_URL is JDBC style connection string for accessing the database. The CCS_DEFAULT_DB_TYPE, CCS_DEFAULT_DB_HOST, CCS_DEFAULT_DB_PORT and CCS_DEFAULT_DB_NAME variables the contain individual values that can be used to established a database connection.
Var variable CCS_VERSION is also defined. It will contain the contents of a file called CCS_VERSION from the root of the application or API repository. If no file is present it will contain 0.0.1.
Environment variables can also be used to pass feature switches to containers. These variables should all be prefixed with CCS_FEATURE_ and contain a value of on or off. For example
The Application/API data S3 bucket is passed in the environment variable: CCS_APP_API_DATA_BUCKET.
The assets bucket is passed in the environment variable: ASSETS_BUCKET.
CCS_FEATURE_EG1=on
These parameters are created by the infrastructure module and the values are stored in the AWS SSM Parameter Store when that module is apply-ed.
The build/xyz pipelines read these values from the SSM Parameter Store and store the values in the ECS Task Definition for the containers when the build/xyz pipeline is apply-ed.
This is important to note as if any of these values change or need to be changed for whatever reason then the infrastructure and build pipelines need to be re-apply-ed.
It is also important to note that as these values are passed to the container using the AWS ECS Task Definition then they will override any environment variable values built into the container image.
Certain entries in the EC2 Parameter store will be automatically turned into environment variables and passed to the running container. These entries can be global that will be passed to all containers or specific to an Application or API.
For a global variable the format is /Environment/global/{Variable Name}
For an Application or API specific variable the format is /Environment/{App or API prefix}/{App or API Name}/{Variable Name}
For example a Parameter Store entry:
/Environment/ccs/cmp/GOOGLE_GEOCODING_API_KEY set to 'QWERTY'
Will result in an environment variable GOOGLE_GEOCODING_API_KEY with the value of 'QWERTY' being available to the running container.
Note that these settings are handled by re-writing the Application or API Dockerfile during the build process. To facilitate this the Dockerfile must contain a special marker to indicate where the additional environment variables should be injected: ##_PARAMETER_STORE_MARKER_##.
For example:
ARG BUILD_TIME
LABEL build_time=$BUILD_TIME
ENV BUILD_TIME=$BUILD_TIME
ARG CCS_VERSION
LABEL ccs_version=$CCS_VERSION
ENV CCS_VERSION=$CCS_VERSION
##_PARAMETER_STORE_MARKER_##
ENV BUILD_PACKAGES curl-dev ruby-dev postgresql-dev build-base tzdata
# Update and install base packages
RUN apk update && apk upgrade && apk add bash $BUILD_PACKAGES nodejs-current-npm gitThese additional parameters are built into the container. For a change to a variables value defined in this way to take effect the application/API build pipeline must executed. This will rebuild the container image and embed the new values. This will require either a change in the source repository, or more simply, the selection of the Release change option from the CodePipeline area of the AWS Console.
It is also important to note that as these values are built into the Docker container image they will be overridden by values passed in via the ECS Task Definition. So environment variables defined in the code pipeline terraform will take precedent over values injected in the container image from the Parameter Store.
The component module used for most build pipelines has a number of parameters that can be used to configure DNS entries and load balancer routing rules for the application. The following parameters are all involved in some way. These are all optional.
- hostname
- path_pattern
- routing_priority_offset
- catch_all
If the hostname parameter is set then this is used specify the sub-domain for accessing the service being deployed. If it is not set then the (mandatory) name parameter is used instead. This is used to create an A record entry in the Route 53 public DNS. This simply maps traffic through to the app application load balancer.
The application load balancer has 2 listeners, one for HTTP and one for HTTPS. The HTTP listener has rules which send redirects to HTTPS to force HTTPS. The HTTPS rules are where the routing to the applications occurs.
If a path_pattern parameter is set then a combined hostname and path based rule is added to the ruleset. In this case no DNS record is registered.
If no path_pattern parameter is specified then then the hostname or name parameter is used as the DNS entry and a simple hostname only rule is added to the rule set.
If services have "overlapping" hostnames - as the crown-marketplace and crown-marketplace-legacy applications do - then the order of the rules is important. The terraform assigns a priority to the rules with hostname/path based rules being set at 1000 + an offset and just hostname rules being set at 5000 + an offset. This ensures that the more specific path based rules come before the less specific hostname only rules. In order to avoid issues with the same priority being used for a rule which is not allowed in AWS the routing_priority_offset parameter is used to define an order. Each component should use a unique value for the routing_priority_offset. The actual value of this is largely arbitrary as the rule order is generally not important - it just needs to be unique.
Each component that is created has an associated AWS target group created. The target group has an associated health check.
Some applications may have a specific health check endpoint. The component health_check_path parameter can be used to specify this. If not specified it defaults to the root of the application or "/".