Skip to content

Installation, Updating and Deleting AWS resources

Brian Riley edited this page Sep 23, 2024 · 20 revisions

<- Home

You will need to install a few things before you're able to build the application. Please note that this system uses Amazon Web Services (AWS) and their CloudFormation technology to build and manage resources like databases and APIs. You will need to create an account if you do not already have one.

Please note that building this application will create resources in your account that will incur charges!

AWS credentials

You must have your AWS credentials setup on your system in order to build the application and infrastructure.

You will also need to install the AWS CLI v2

AWS resource prerequisites

You will need to create or identify a Virtual Private Cloud (VPC) and both private and public Subnets to house the resources built by this repository.

If you are unfamiliar with setting up VPCs, then we highly recommend getting in touch with your networking team for guidance on how to properly configure it's security settings

You will also need to have a domain name for your system and create a Hosted Zone for that domain so that the SSL Certificate and Route53 records can be constructed.

We recommend building the above resources with CloudFormation so that you can export the VPC Id, Subnet Ids and Hosted Zone Id as stack outputs so that they are available to the templates in this repository. If they were not built by Cloud Formation, then you can place them in SSM parameters and update the config/[env]/*.yaml files to fetch them from SSM instead of an existing stack (see below for more info on the necessary SSM parameters).

These Ids should all be available as 'Outputs' of a CloudFormation template. You may need to update the [env]/[context]/config.yaml files to point to the correct external CloudFormation stacks.

If the values are not available in the CloudFormation stack, then you should update the initial_setup.rb file so that you can pass these ids in and have them added to your SSM parameter store. If you do this, you will need to update the [env]/[context]/config.yaml files along with any Lambda layer/function build scripts defined in the DMSP Prototype - API repository

Sceptre installation and setup

This repository uses Sceptre to help manage and orchestrate CloudFormation templates. To install it you should:

  • Ensure Python is installed: python --version
  • Assuming Python is installed, you must then install pipenv (Python package mamnager): pip install --user pipenv
  • Then install Sceptre: pip install sceptre
  • Make sure the install was successful: sceptre --version

You can update Sceptre (at any time) with the following command: pip install sceptre -U

Install UC3 Sceptre utilities

UC3 manages a Sceptre utilities library that provides resolvers and hooks to make tasks easier. To install the utilities:

Required SSM parameters

The following SSM parameters must be setup manually in your AWS account prior to installation. We do not recommend scripting these particular parameters because they contain sensitive information that should not be pushed into this (or your fork's) repository.

Note that the instructions below use the AWS CLI but you may set them up through the AWS console as well.

You can run ruby initial_setup.rb -h to see the options available for setting the below SSM parameters OR run them individually as defined below.

For example:

> ruby initial_setup.rb -h                                                                                                                        
Usage: ruby initial_setup [options]
    -e, --environment ENV            The environment (Example: dev)
    -r, --region AWS_REGION          The AWS region (Default: us-west-2)
    -o, --override                   Replace existing values

    -h, --hosted-zone ZONE           The HostedZoneId (e.g. BUG934GKJH88)
    -d, --domain-name DOMAIN         The Domain Name of the application (e.g. my.site.edu)

    -w, --admin-email EMAIL          The Administrator email address (e.g. helpdesk@example.com)

    -a, --ezid-api-url URL           The EZID API URL (e.g. https://ezid-stg.cdlib.org/)
    -b, --ezid-base-url URL          The EZID DOI URL (e.g. https://doi.org/)
    -I, --ezid-hosting-institution   How EZID displays your name on DOIs (e.g. University of Example)
    -s, --ezid-shoulder SHOULDER     Your EZID DOI shoulder (Example: 10.00000/Z9)
    -u, --ezid-username USER         Your EZID username (Example: player-one)
    -p, --ezid-password PWD          Your EZID password (Example: Pl@y3rOne)
    -z, --ezid-paused true|false     Whether or not the integration with EZID is paused

    -c, --cache-secret TOKEN_HASH_SECRET    The secret used by the DMPTool backend for hashing tokens before they're stored in the cache
    -j, --jwt-secret JWT_SECRET      The JSON Web Token secret used by the DMPTool frontend and backend for access tokens
    -x, --jwt-refresh-secret JWT_REFRESH_SECRET  The JSON Web Token secret used by the DMPTool frontend and backend for refresh tokens

    -m, --db-password PWD            Your RDS database password (Example: passw0rd12E)

> ruby initial_setup.rb -e dev -r us-west-2 -a my-email@example.com -x -m -n 'My Organization Name' -s 10.00000/Z9 -u username -p password -m dbpassword  -d example.com -w helpdesk@example.com -j 1234567890abcdefghijklmnop

The following SSM Parameters are often defined at a global level or because they resource that they are associated with cannot be created via CloudFormation. If they are not defined, you will need to create them:

  • /uc3/CodeStarConnectionArn <-- Used to provide access to GitHub repositories for CodePipeline
  • /uc3/MainAccountBackupVault <-- Used to backup S3 buckets and RDS
  • /uc3/prd/CfnServiceRoleArn` <-- The CloudFormation service role to assume when running some commands

If you prefer to add the SSM Parameters individually you can copy paste the following commands:

Route53 HostedZoneId

Add your HostedZone id to the SSM parameter store.

  • Admin email: aws ssm put-parameter --name /uc3/dmp/hub/[env]/HostedZoneId --value "[id]" --type "String"

Administrator Email

Add your administrator email to the SSM parameter store. This email will receive fatal error messages produced by the Lambdas

  • Admin email: aws ssm put-parameter --name /uc3/dmp/hub/[env]/AdminEmail --value "[email]" --type "String"

Note: After the sqs.yaml stack is created, AWS will send an email addresses you define here. You will need to click the link in that email to confirm the subscription!

EZID Credentials

The following SSM parameters should be defined for integration with EZID for registering DOIs:

  • EZID api url (e.g. https://ezid-stg.cdlib.org/): aws ssm put-parameter --name /uc3/dmp/hub/[env]/EzidApiUrl --value "[url]" --type "String"
  • EZID base url (the prefix for the DOI - e.g. https://doi.org/): aws ssm put-parameter --name /uc3/dmp/hub/[env]/EzidBaseUrl --value "[url]" --type "String"
  • EZID debug mode: aws ssm put-parameter --name /uc3/dmp/hub/[env]/EzidDebugMode --value "true" --type "String"
  • EZID pause flag: aws ssm put-parameter --name /uc3/dmp/hub/[env]/EzidPaused --value "false" --type "String"
  • EZID hosting institution: aws ssm put-parameter --name /uc3/dmp/hub/[env]/EzidHostingInstitution --value "[name]" --type "String"
  • EZID password: aws ssm put-parameter --name /uc3/dmp/hub/[env]/EzidPassword --value "[password]" --type "SecureString"
  • EZID shoulder: aws ssm put-parameter --name /uc3/dmp/hub/[env]/EzidShoulder --value "[shoulder]" --type "String"
  • EZID username: aws ssm put-parameter --name /uc3/dmp/hub/[env]/EzidUsername --value "[username]" --type "SecureString"

Note: The 'EzidDebugMode' boolean instructs the EzidPublisher Lambda to skip the process of contacting EZID. This can be used when testing DMP record creation. The 'EzidPaused' boolean instructs the EzidPublisher Lambda to stash the work into the EventBus' Archive so that it can be processed at a later date. This is typically used when EZID is going to be offline for a maintenance period.

DMPTool System Variables

The following SSM parameters should be defined to support the DMPTool nextJS frontend and Apollo Server backend. These variables persist beyond the lifetime of any given container instance.

  • TOKEN_HASH_SECRET: aws ssm put-parameter --name /uc3/dmp/tool/[env]/CacheHashSecret --value "[value]" --type "SecureString"
  • JWT_SECRET: aws ssm put-parameter --name /uc3/dmp/tool/[env]/JWTSecret --value "[value]" --type "SecureString"
  • JWT_REFRESH_SECRET: aws ssm put-parameter --name /uc3/dmp/tool/[env]/JWTRefreshSecret --value "[value]" --type "SecureString"

External Systems

If you have any external systems that will be communicating with the DMPHub API, you will need to add any OAuth client credentials required by that external system. This will be used when attempting to download DMP PDF documents and/or send updated metadata back to the external system.

__Note that the system_key in the examples below MUST match the PK value for the external system in the Dynamo table. (e.g. the PROVENANCE#example DyanmoDB item would result in an ssm parameter of /uc3/dmp/hub/[env]/example/client_id). See below for an example Provenance item.

  • Client id: aws ssm put-parameter --name /uc3/dmp/hub/[env]/[system_key]/client_id --value "[username]" --type "SecureString"
  • Client secret: aws ssm put-parameter --name /uc3/dmp/hub/[env]/[system_key]/client_secret --value "[username]" --type "SecureString"

You can use aws ssm get-parameters-by-path --path '/uc3/dmp/hub/[env]/' to see what parameters have already been set.

Any new parameters should maintain the /uc3/dmp/hub/[env]/ prefix!

Modifying the Sceptre config

Sceptre uses configuration YAML files that work in conjunction with the CloudFormation templates. They allow us to pull in variables from various locations (e.g. SSM, other CloudFormation stacks, etc.) and inject them into the CloudFormation templates as parameters.

The config/config.yaml defines the name of your system project_code: uc3-dmp-hub which will be used to name your CloudFormation stacks and all of the resources that are created. Change this to suit your needs. It also defines the AWS region you want to work in region: us-west-2 change this to a more suitable region if necessary.

The config/[env]/config.yaml defines variables for the specific environment you are working with. Please review and update this file as needed.

Note that if you needed to add your VPC, Subnet and/or HostedZone Ids to SSM, then you will need to change how they are relieved in this file. For example: vpc_id: !stack_output_external cdl-uc3-dev-vpc-stack::vpc would need to become vpc_id: !ssm_parameter /uc3/dmp/hub/[env]/VpcId

Building/Updating/Deleting the system

Sceptre manages the infrastructure in a hierarchically order. This means that if you try to create a stack whose config file references outputs from another stack, then Sceptre will first try to create the referenced stack(s). When deleting, it does the reverse.

For example if you issue a sceptre create dev/cloudfront.yaml, Sceptre will first create cert.yaml, waf.yaml, etc.. If you issue a sceptre delete dev/cert.yaml, Sceptre will first delete cloudfront.yaml but will leave waf.yaml alone.

Creating

Once all of the above steps have been completed, you can build the DMPHub system.

If this is your first time building the system, then run the following to build out the entire stack:

  • sceptre create [env]/regional/sqs.yaml builds out the SQS Queue and SNS topics that will be used by the Lambdas to send fatal error emails to the AdminEmail you defined in SSM and the CodeBuild to email the results of the build to the CodeBuildEmail you defined in SSM.
  • sceptre create [env]/regional/eventbridge.yaml builds out the custom EventBus and Archive which are used by the Lambdas to send messages to other Lambdas (e.g. postDmps Lambda with send an event that triggers the EzidPublisher Lambda).
  • sceptre create [env]/regional/cognito.yaml builds out the Cognito UserPool and Cognito clients so that they can access the API Gateway created in the next step. It will also trigger the build for the ACM cert, WAF, route53 and CloudFront resources.
  • sceptre create [env]/regional/dynamo.yaml builds the Dynamo Table and then kicks off the AWS SAM build and deploy for the Lambdas and API Gateway resources as well as the Swagger API documentation.

Note that if a stack already exists, Sceptre will skip that stack.

If you need to create a stack individually just specify that template. For example: sceptre create [env]/[regional|global]/[template].yaml

Updating

If the system has already been built and you want to apply a change to one of the Sceptre managed resources you should run sceptre update [env]/[region_type]/[config].yaml.

Deleting

If you want to delete resources then run sceptre delete [env]/[region_type]/[config].yaml

Note EC2 Security Groups are difficult to delete in an automated way. You will likely need to manually disconnect the Lambdas, CodeBuilds and RDS security groups from their associated resources before they can be deleted by CloudFormation!