Add lambda to bootstrap tower database #16

tthyer · 2021-06-24T19:01:55Z

This adds a Lambda to perform some bootstrapping statements on the tower database in the Aurora DB cluster.
https://sagebionetworks.jira.com/browse/WORKFLOWS-12

zaro0508

Another suggestion that's not part of this change.. Would it make sense to replace your ClusterSettings map with AWS::RDS::DBClusterParameterGroup[1]

[1] https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-rds-dbclusterparametergroup.html

zaro0508 · 2021-06-28T17:02:44Z

templates/nextflow-aurora-mysql.yaml

@@ -125,6 +152,137 @@ Resources:
      VpcSecurityGroupIds:
        - !Ref ClusterSecurityGroup

+  # this custom resource exists to trigger the bootstrap lambda on creation
+  BootstrapLambdaTrigger:


Suggestion but not required.. This resource will create a lambda instance for every new DB instance. Would it make sense to decouple the lambda. Install one lambda and reuse it to setup each instance of a DB.

If this were going to be used more than once, I would say that's the thing to do. I expect this to be used only once per account. Creating a lambda to run a couple of SQL statements is already overkill, and I'd prefer not to over-optimize when there is almost no chance that we would create more than one.

zaro0508 · 2021-06-28T19:38:20Z

templates/nextflow-aurora-mysql.yaml

@@ -65,6 +65,18 @@ Resources:
        ExcludeCharacters: '"@/\$`&'
        PasswordLength: 16

+  AuroraTowerUserSecret:


would it make sense to change Tower to a more generic name in case nexflow change's it's DB user? something like AuroraNextflowUserSecret would make it agnostic to a specific Tower username. That way you can just pass in any username you want.

Sure, I can name it something like AuroraNextflowTowerDatabaseUserSecret

zaro0508 · 2021-06-28T19:40:49Z

templates/nextflow-aurora-mysql.yaml

+            try:
+              secret_response = boto3.client('secretsmanager').get_secret_value(SecretId=environ['Tower_User_Secret_ARN'])
+              secret_string = secret_response['SecretString']
+              alter_db_response = submit_sql("ALTER DATABASE tower CHARACTER SET utf8 COLLATE utf8_bin;")


would it make sense to parameterize db name?

sure, might as well

ah actually I already did that; just missed replacing the string here

zaro0508 · 2021-06-28T19:41:30Z

templates/nextflow-aurora-mysql.yaml

+              secret_string = secret_response['SecretString']
+              alter_db_response = submit_sql("ALTER DATABASE tower CHARACTER SET utf8 COLLATE utf8_bin;")
+              log.debug(f"Alter DB response: {alter_db_response}")
+              create_user_response = submit_sql("CREATE USER 'tower' IDENTIFIED BY '" + secret_string + "';")


per above suggestion, instead of the static tower db username maybe you can parameterize it and allow passing in any username?

tthyer · 2021-06-29T00:07:23Z

@zaro0508 regarding the question #16 (review)
No -- these sort of parameter groups have a very narrow use to do with configuration a database engine.
We are actually already using one, it just happens to be the default, so it doesn't appear in the stack

zaro0508 · 2021-06-29T16:29:54Z

templates/nextflow-aurora-mysql.yaml

-      Name: !Sub '${AWS::StackName}-TowerUserSecret'
-      Description: !Sub 'Aurora MySQL Tower User Secret for CloudFormation Stack ${AWS::StackName}'
+      Name: !Sub '${AWS::StackName}-NextflowTowerDatabaseUserSecret'
+      Description: !Sub 'Aurora MySQL Nextflow Tower User Secret for CloudFormation Stack ${AWS::StackName}'


shouldn't this be

!Sub 'Aurora MySQL ${DBUserName} User Secret for CloudFormation Stack ${AWS::StackName}'

zaro0508 · 2021-06-29T16:35:42Z

templates/nextflow-aurora-mysql.yaml

+              log.debug(f"Grant privileges response: {grant_response}")
+              return {
+                'status': cfnresponse.SUCCESS,
+                'message': 'Succeeded bootstrapping tower database',


tower/{db_name}

zaro0508 · 2021-06-29T16:35:49Z

templates/nextflow-aurora-mysql.yaml

+              log.error(f'An error was encountered when bootstrapping: {e}')
+              return {
+                'status': cfnresponse.FAILED,
+                'message': 'Failed bootstrapping tower database',


tower/{db_name}

Add lambda to bootstrap tower database

4af90ce

tthyer force-pushed the workflows-12_bootstrap_aurora branch from 289f22e to 4af90ce Compare June 24, 2021 19:02

tthyer requested review from BrunoGrandePhD, daisyhan97, thomasyu888 and zaro0508 June 24, 2021 19:04

zaro0508 requested changes Jun 28, 2021

View reviewed changes

tthyer requested a review from zaro0508 June 29, 2021 00:08

zaro0508 approved these changes Jun 29, 2021

View reviewed changes

zaro0508 reviewed Jun 29, 2021

View reviewed changes

Some further parameterization and renaming

7d12e34

tthyer force-pushed the workflows-12_bootstrap_aurora branch from 9b1afee to 7d12e34 Compare June 29, 2021 18:29

tthyer merged commit 2eea2ed into main Jun 29, 2021

tthyer deleted the workflows-12_bootstrap_aurora branch June 29, 2021 18:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add lambda to bootstrap tower database #16

Add lambda to bootstrap tower database #16

tthyer commented Jun 24, 2021 •

edited

Loading

zaro0508 left a comment

zaro0508 Jun 28, 2021

tthyer Jun 28, 2021 •

edited

Loading

zaro0508 Jun 28, 2021

tthyer Jun 28, 2021

zaro0508 Jun 28, 2021

tthyer Jun 28, 2021

tthyer Jun 28, 2021

zaro0508 Jun 28, 2021

tthyer Jun 28, 2021

tthyer commented Jun 29, 2021

zaro0508 Jun 29, 2021

zaro0508 Jun 29, 2021

zaro0508 Jun 29, 2021

Add lambda to bootstrap tower database #16

Add lambda to bootstrap tower database #16

Conversation

tthyer commented Jun 24, 2021 • edited Loading

zaro0508 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tthyer Jun 28, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tthyer commented Jun 29, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tthyer commented Jun 24, 2021 •

edited

Loading

tthyer Jun 28, 2021 •

edited

Loading