Dojo docker image with tools needed to deploy infrastructure in scope of deductions team.
Tested and released images are published to dockerhub as nhsdev/deductions-infra-dojo
- Setup docker.
- Install Dojo binary.
- Provide a Dojofile at the root of the project:
DOJO_DOCKER_IMAGE="nhsdev/deductions-infra-dojo:<commit>"
- Create and enter the container by running
dojo
at the root of project.
By default, current directory in docker container is /dojo/work
.
This repository imports shared AWS helpers from prm-deductions-support-infra.
They can be found utils
directory after running any task from tasks
file.
In order to get sufficient access to work with terraform or AWS CLI:
Make sure to unset the AWS variables:
unset AWS_ACCESS_KEY_ID
unset AWS_SECRET_ACCESS_KEY
unset AWS_SESSION_TOKEN
The following set-up is based on the README of assume-role tool
Setup a profile for each role you would like to assume in ~/.aws/config
.
As an example:
[profile default]
region = eu-west-2
output = json
[profile admin]
region = eu-west-2
role_arn = <role-arn>
mfa_serial = <mfa-arn>
source_profile = default
Your source_profile
needs to match your profile in ~/.aws/credentials
.
[default]
aws_access_key_id = <your-aws-access-key-id>
aws_secret_access_key = <your-aws-secret-access-key>
brew install remind101/formulae/assume-role
Run the following command with the profile configured in your ~/.aws/config
:
assume-role admin
Run the following command with the profile configured in your ~/.aws/config
:
eval $(dojo "echo <mfa-code> | assume-role admin")
Run the following command to confirm you assumed the correct role:
aws sts get-caller-identity
Work with terraform as per usual.
When creating the new ssm keys, please follow the agreed convention as per the design specified below:
- all parts of the keys are lower case
- the words are separated by dashes (
kebab case
) env
is optional
Please follow this design to ensure the ssm keys are easy to maintain and navigate through:
Type | Design | Example |
---|---|---|
User-specified | /repo/<env>?/user-input/ |
/repo/${var.environment}/user-input/db-username |
Auto-generated | /repo/<env>?/output/<name-of-git-repo>/ |
/repo/output/prm-deductions-base-infra/root-zone-id |
- Gain access to AWS as described above
- Generate VPN client configuration for each environment:
NHS_ENVIRONMENT=<env-name> ./tasks generate_vpn_client_crt <your-first-name.your-last-name>
E.g. to get access to both dev and test environments, you'll need to run:
NHS_ENVIRONMENT=dev ./tasks generate_vpn_client_crt <your-first-name.your-last-name>
NHS_ENVIRONMENT=test ./tasks generate_vpn_client_crt <your-first-name.your-last-name>
- Open the ovpn file that you want to revoke access for. This can be found in the directory where it was created: prm-deductions-infra → client-config. You may have an ovpn file per environment and will be named something like dev.john.smith.ovpn
- Copy the certificate from the ovpn - this is everything between the tags. If someone else is revoking your certificates for you, send this certificate to them securely.
- Create a new file in the prm-deductions-infra → utils directory, named similarly to your ovpn file but ending with .crt instead - dev.john.smith.crt
- Paste your copied certificate into this new file
- Gain access to AWS as described above
- Revoke VPN client configuration for each environment:
NHS_ENVIRONMENT=<env-name> ./tasks revoke_vpn_client_crt <your-first-name.your-last-name>
All vpn endpoints use the same CA which needs to be generated only once.
The operation is automated, mostly for reference:
./tasks generate_vpn_ca
Each VPN endpoint has its own server certificate signed by above CA. The operation is automated in the task:
NHS_ENVIRONMENT=dev ./tasks generate_vpn_server_crt
It is part of a pipeline that deploys each environment.
When VPN CA is changed, the old certificate is still being used by the VPN endpoint. As a result, running the command NHS_ENVIRONMENT=dev ./tasks generate_vpn_server_crt
to generate a developer’s VPN certificate won’t create a successful VPN connection. The VPN endpoint needs to be deleted or modified not to use the old certificate.
If you need to delete the VPN endpoint please use terraform.
Please note that when the VPN endpoint is deleted in the AWS console, it may cause some sync-up issues with the terraform state as terraform is still looking for the previous VPN endpoint ID and its resources.
The newly created certificate should be associated with the VPN endpoint. If you deleted VPN endpoint, the terraform will handle that association. Otherwise you need to modify the VPN endpoint to use new certificate in the AWS console.
After the deletion of the old VPN certificate, each VPN certificate should be recreated locally and the old certificates should be replaced. A developer can run NHS_ENVIRONMENT=dev ./tasks generate_vpn_server_crt
locally to generate the new VPN certificate.
For prod environment only there is a certificate revocation list (CRL) which could expire, causing a TLS error, TLS handshake failed as documented here:
https://docs.aws.amazon.com/vpn/latest/clientvpn-admin/troubleshooting.html#client-cannot-connect
In case anyone is experiencing the issue, run the Revoking VPN client keys step above. You can do this targeting your own certs (which you'll have to regenerate via Generating VPN client keys).
They will be also stored in a Repo folder in CloudWatch Insights. More information here: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_Insights-Saving-Queries.html
###Tabular Views - enable to see service in the separate CloudWatch column:
fields @timestamp, @message, service
| filter @message not like /(H|h)ealth/
| sort @timestamp desc
###Filter by severity levels - useful if we would like to see all the errors:
fields @timestamp, @message, service
| filter @message not like /(H|h)ealth/
| filter @message like /(ERROR|error|Error)/
| filter @message like /500/
| sort @timestamp desc
###Dev - non-compliant messages
fields @timestamp, @message, @logStream
| filter not ispresent(service) or not ispresent(environment) or not ispresent(level)
| sort @timestamp desc
###Log streams by non-compliance
fields @timestamp, @message, @logStream
| filter not ispresent(service) or not ispresent(environment) or not ispresent(level)
| stats count(*) by @logStream
###Test - All level messages above DEBUG
fields @timestamp, level, service, @message
| filter level != 'DEBUG'
| filter @message not like /(H|h)ealth/
| sort @timestamp desc
###Test - Interactions with queue raw-inbound
fields @timestamp, @message, service, correlationId
| filter queue = 'raw-inbound'
| sort @timestamp desc
To get the repo value metrics calculations from prod, do following (updating week-ending date appropriately):
eval $(assume-role prod)
NHS_ENVIRONMENT=prod ./tasks get_value_metrics "2023-01-27"
This script currently assumes your aws profile for production is called prod
.
There are two types of api keys in use in the Repository team services. They are maintained by manual update of certain
SSM parameters and running of the service-api-keys
and user-api-keys
pipelines.
These are for non-user clients to access a service, including access from other services. The list of service-client
api key pairs is specified in an env-specific list in SSM (.../service-api-keys
).
The format of the list is of full service api key SSM parameter keys for that environment. It can be newline-delimited instead of comma-delimited for readability.
For improved readability - esp. considering this is access security config - moving to a more pared-down JSON format might be appropriate.
These are for individual user access to services. The list of users that user api keys will be generated for is an
env-specific parameter in SSM (.../repo-dev-list
).
When the service-api-keys
or user-api-keys
pipelines run they will generate any missing keys and remove ones not in
their related initial list.
Setting RotateApiKey
tag on one of these SSM parameters will cause it to be rotated when the pipeline runs, but looks
like that shouldn't be done if there are any outstanding add/deletes as generation runs in parallel to rotation.