Creating a secret with SecretRotationApplication.POSTGRES_ROTATION_SINGLE_USER doesn't seem to work #6358

thekevinbrown · 2020-02-19T08:16:01Z

I create a Secret with rotation for my application database credentials like this:

// Create a rotating secret we can use to access the database.
const secret = new DatabaseSecret(construct, `consumer-api-db`, {
	username: 'consumer-api',
});

secret.attach(database);
secret.grantRead(consumerApi); // My Lambda

const databaseSecretRotation = new SecretRotation(
	this,
	`consumer-api-rotation`,
	{
		secret: secret,
		application: SecretRotationApplication.POSTGRES_ROTATION_SINGLE_USER,
		vpc,
		vpcSubnets: vpc.selectSubnets({ subnetType: SubnetType.PRIVATE }),
		target: database,
		automaticallyAfter: Duration.days(30),
	}
);

But when I go into Secrets Manager and click "Rotate Immediately" I get this in the logs:

Database engine must be set to 'postgres' in order to use this rotation lambda

And when I try to use the secret, I get a password error because the rotation hasn't happened.

Reproduction Steps

See Above

Error Log

Database engine must be set to 'postgres' in order to use this rotation lambda

Environment

CLI Version : 1.24.0
Framework Version: 1.24.0
OS : OS X 10.15.3
Language : TypeScript

Other

How do I correctly create a single user rotation for that secret?

This is 🐛 Bug Report

The text was updated successfully, but these errors were encountered:

jogold · 2020-02-19T08:57:12Z

You need to work with the attached secret (which contains DB informations):

const attachedSecret = secret.attach(database);

const databaseSecretRotation = new SecretRotation(
	this,
	`consumer-api-rotation`,
	{
		secret: attachedSecret, # use attached secret
		application: SecretRotationApplication.POSTGRES_ROTATION_SINGLE_USER,
		vpc,
		vpcSubnets: vpc.selectSubnets({ subnetType: SubnetType.PRIVATE }),
		target: database,
		automaticallyAfter: Duration.days(30),
	}
);

See https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-secretsmanager-secrettargetattachment.html

You can also work from the database with addRotationSingleUser() which handles this for you.

thekevinbrown · 2020-02-19T09:34:49Z

@jogold, thank you for the very prompt response.

Unfortunately even after deleting the whole stack and recreating it using the attached version of the secret, I'm still getting 'Database engine must be set to 'postgres' in order to use this rotation lambda' when I run the rotation.

jogold · 2020-02-19T09:36:08Z

What is the content of the json in the secret?

thekevinbrown · 2020-02-19T09:44:19Z

{
  "password": "[random value]",
  "dbname": "[my database name]",
  "engine": "postgres",
  "port": 5432,
  "host": "[the host for the database server]",
  "username": "consumer-api"
}

Edit: I'm going to double check that the secret ARN matches with the logs, because in the Lambda logs I'm seeing it successfully pull down the secret, and I don't understand how it'd give this error with that secret.

thekevinbrown · 2020-02-19T10:22:29Z

Ok, I'm really not sure what's going on. It looks like the lambda on first run creates the secret successfully, and then fails on subsequent runs.

This is an Aurora Postgres Serverless cluster if that makes a difference.

Here are the logs from creation:

  | 2020-02-19T21:18:42.269+11:00 | START RequestId: 0abae0b1-e42f-4b3f-9073-53e8f868f149 Version: $LATEST
  | 2020-02-19T21:18:42.533+11:00 | [INFO]	2020-02-19T10:18:42.533Z	0abae0b1-e42f-4b3f-9073-53e8f868f149	Found credentials in environment variables.
  | 2020-02-19T21:18:44.165+11:00 | [INFO]	2020-02-19T10:18:44.165Z	0abae0b1-e42f-4b3f-9073-53e8f868f149	createSecret: Successfully put secret for ARN arn:aws:secretsmanager:eu-west-1:417241902280:secret:consumerapidbaccess5605D472-6o2iS1279x4h-b9cxBz and version 48e7c167-9049-4892-af86-84f3e3b85e26.
  | 2020-02-19T21:18:44.171+11:00 | END RequestId: 0abae0b1-e42f-4b3f-9073-53e8f868f149
  | 2020-02-19T21:18:44.171+11:00 | REPORT RequestId: 0abae0b1-e42f-4b3f-9073-53e8f868f149	Duration: 1897.22 ms	Billed Duration: 1900 ms	Memory Size: 128 MB	Max Memory Used: 73 MB	Init Duration: 204.34 ms
  | 2020-02-19T21:18:44.198+11:00 | START RequestId: 8b7fe42a-137f-426a-8537-b8fde0884884 Version: $LATEST
  | 2020-02-19T21:18:44.940+11:00 | "Database engine must be set to 'postgres' in order to use this rotation lambda": KeyErrorTraceback (most recent call last):  File "/var/task/lambda_function.py", line 76, in lambda_handler    set_secret(service_client, arn, token)  File "/var/task/lambda_function.py", line 160, in set_secret    conn = get_connection(get_secret_dict(service_client, arn, "AWSPREVIOUS"))  File "/var/task/lambda_function.py", line 313, in get_secret_dict    raise KeyError("Database engine must be set to 'postgres' in order to use this rotation lambda")KeyError: "Database engine must be set to 'postgres' in order to use this rotation lambda"
  | 2020-02-19T21:18:44.952+11:00 | END RequestId: 8b7fe42a-137f-426a-8537-b8fde0884884
  | 2020-02-19T21:18:44.952+11:00 | REPORT RequestId: 8b7fe42a-137f-426a-8537-b8fde0884884	Duration: 751.14 ms	Billed Duration: 800 ms	Memory Size: 128 MB	Max Memory Used: 75 MB
  | 2020-02-19T21:20:22.843+11:00 | START RequestId: 2358cfb8-d071-4165-8e83-8fb7d145cf71 Version: $LATEST
  | 2020-02-19T21:20:23.199+11:00 | [INFO]	2020-02-19T10:20:23.199Z	2358cfb8-d071-4165-8e83-8fb7d145cf71	createSecret: Successfully retrieved secret for arn:aws:secretsmanager:eu-west-1:417241902280:secret:consumerapidbaccess5605D472-6o2iS1279x4h-b9cxBz.
  | 2020-02-19T21:20:23.251+11:00 | END RequestId: 2358cfb8-d071-4165-8e83-8fb7d145cf71
  | 2020-02-19T21:20:23.251+11:00 | REPORT RequestId: 2358cfb8-d071-4165-8e83-8fb7d145cf71	Duration: 387.52 ms	Billed Duration: 400 ms	Memory Size: 128 MB	Max Memory Used: 76 MB
  | 2020-02-19T21:20:23.260+11:00 | START RequestId: b792bc56-d780-4d28-ab17-421d9df7d677 Version: $LATEST
  | 2020-02-19T21:20:23.777+11:00 | "Database engine must be set to 'postgres' in order to use this rotation lambda": KeyErrorTraceback (most recent call last):  File "/var/task/lambda_function.py", line 76, in lambda_handler    set_secret(service_client, arn, token)  File "/var/task/lambda_function.py", line 160, in set_secret    conn = get_connection(get_secret_dict(service_client, arn, "AWSPREVIOUS"))  File "/var/task/lambda_function.py", line 313, in get_secret_dict    raise KeyError("Database engine must be set to 'postgres' in order to use this rotation lambda")KeyError: "Database engine must be set to 'postgres' in order to use this rotation lambda"
  | 2020-02-19T21:20:23.792+11:00 | END RequestId: b792bc56-d780-4d28-ab17-421d9df7d677
  | 2020-02-19T21:20:23.792+11:00 | REPORT RequestId: b792bc56-d780-4d28-ab17-421d9df7d677	Duration: 529.89 ms	Billed Duration: 600 ms	Memory Size: 128 MB	Max Memory Used: 77 MB

thekevinbrown · 2020-02-19T11:07:00Z

My working hypothesis is:

I haven't yet created this user in the database. It's not the master credentials, so logging in will fail.
When that happens, the lambda will try the previous credentials here.
When it gets back an empty dictionary there, it complains that the engine key is missing.

So, if that's correct the solution is to create the user with the password that the lambda is expecting. I was creating it with a random password I generated because I (incorrectly) assumed the rotation would work with the database master secret to set the user's password, not the user's password itself.

jogold · 2020-02-19T11:15:46Z

The single user rotation scheme is supposed to be used with the master credentials.

The error you get comes from this line:
https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/blob/9f0707343521d2e7440f3ebf14ec8bd0fa6129e4/SecretsManagerRDSPostgreSQLRotationSingleUser/lambda_function.py#L313

Not sure this is a CDK issue.

thekevinbrown · 2020-02-19T11:42:09Z

Yup, that fixed it, just needed to create the user with the current secret value as the password so the rotation could log in as that user.

If rotations aren't meant to be used with other user accounts the docs here should be updated as they seem to say otherwise:

Note: This user must be created manually in the database using the master credentials. The rotation will start as soon as this user exists.

Happy to flag the confusing error message wherever those lambdas are maintained. Could you let me know where I should create that issue?

jogold · 2020-02-19T11:49:54Z

If rotations aren't meant to be used with other user accounts the docs here should be updated as they seem to say otherwise:

Normally with other users accounts you use the multi user scheme, more info here https://docs.aws.amazon.com/secretsmanager/latest/userguide/rotating-secrets-one-user-one-password.html and here https://docs.aws.amazon.com/secretsmanager/latest/userguide/rotating-secrets-two-users.html

Happy to flag the confusing error message wherever those lambdas are maintained. Could you let me know where I should create that issue?

This might be the right repo https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas

thekevinbrown · 2020-02-19T12:58:27Z

@jogold,

That makes sense, but that's not working either. I updated to:

	const attachedSecret = secret.attach(database);

	const databaseSecretRotation = new SecretRotation(construct, `consumer-api-rotation`, {
		secret: attachedSecret,
		masterSecret: database.secret,
		application: SecretRotationApplication.POSTGRES_ROTATION_MULTI_USER,
		vpc,
		vpcSubnets: vpc.selectSubnets({ subnetType: SubnetType.PRIVATE }),
		target: database,
		automaticallyAfter: Duration.days(30),
	});

and I'm getting:

'masterarn': KeyErrorTraceback (most recent call last):  File "/var/task/lambda_function.py", line 79, in lambda_handler    set_secret(service_client, arn, token)  File "/var/task/lambda_function.py", line 172, in set_secret    master_arn = current_dict['masterarn']KeyError: 'masterarn'

after I destroy the stack, create it, then create the user with the credentials in the secret.

jogold · 2020-02-19T13:24:51Z

You have to "manually" add the masterarn key in your user secret, the CloudFormation attachment will not do that for you.

Something like:

const userSecret = new Secret(this, 'UserSecret', {
  generateSecretString: {
    secretStringTemplate: JSON.stringify({
      username: 'your-user-name',
      masterarn: database.secret.secretArn
    }),
   generateStringKey: 'password'
  }
});
const attachedSecret = userSecret.attach(database);

Then your SecretRotation and when the stack is deployed you create the user secret in the DB.

aws-cdk/packages/@aws-cdk/aws-secretsmanager/lib/secret-rotation.ts

Line 142 in d4a132b

    
              *   "masterarn": <required for multi user rotation: the arn of the master secret which will be used to create users/change passwords>

thekevinbrown · 2020-02-20T01:10:31Z

Exactly what I needed, thank you @jogold, really appreciate your help.

Add an optional `masterSecret` prop in `DatabaseSecret` and clarify documentation for the multi user rotation schema. Related to aws#6358

Add an optional `masterSecret` prop in `DatabaseSecret` and clarify documentation for the multi user rotation schema. Related to #6358 Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

thekevinbrown added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Feb 19, 2020

SomayaB added the @aws-cdk/aws-secretsmanager Related to AWS Secrets Manager label Feb 19, 2020

SomayaB assigned skinny85 Feb 19, 2020

SomayaB removed the needs-triage This issue or PR still needs to be triaged. label Feb 19, 2020

thekevinbrown closed this as completed Feb 20, 2020

jogold added a commit to jogold/aws-cdk that referenced this issue Feb 24, 2020

feat(rds): master secret in DatabaseSecret

b254c42

Add an optional `masterSecret` prop in `DatabaseSecret` and clarify documentation for the multi user rotation schema. Related to aws#6358

jogold mentioned this issue Feb 24, 2020

feat(rds): optional master secret in DatabaseSecret #6415

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Creating a secret with SecretRotationApplication.POSTGRES_ROTATION_SINGLE_USER doesn't seem to work #6358

Creating a secret with SecretRotationApplication.POSTGRES_ROTATION_SINGLE_USER doesn't seem to work #6358

thekevinbrown commented Feb 19, 2020 •

edited

Loading

jogold commented Feb 19, 2020 •

edited

Loading

thekevinbrown commented Feb 19, 2020

jogold commented Feb 19, 2020

thekevinbrown commented Feb 19, 2020 •

edited

Loading

thekevinbrown commented Feb 19, 2020 •

edited

Loading

thekevinbrown commented Feb 19, 2020

jogold commented Feb 19, 2020

thekevinbrown commented Feb 19, 2020

jogold commented Feb 19, 2020

thekevinbrown commented Feb 19, 2020 •

edited

Loading

jogold commented Feb 19, 2020 •

edited

Loading

thekevinbrown commented Feb 20, 2020

Creating a secret with SecretRotationApplication.POSTGRES_ROTATION_SINGLE_USER doesn't seem to work #6358

Creating a secret with SecretRotationApplication.POSTGRES_ROTATION_SINGLE_USER doesn't seem to work #6358

Comments

thekevinbrown commented Feb 19, 2020 • edited Loading

Reproduction Steps

Error Log

Environment

Other

jogold commented Feb 19, 2020 • edited Loading

thekevinbrown commented Feb 19, 2020

jogold commented Feb 19, 2020

thekevinbrown commented Feb 19, 2020 • edited Loading

thekevinbrown commented Feb 19, 2020 • edited Loading

thekevinbrown commented Feb 19, 2020

jogold commented Feb 19, 2020

thekevinbrown commented Feb 19, 2020

jogold commented Feb 19, 2020

thekevinbrown commented Feb 19, 2020 • edited Loading

jogold commented Feb 19, 2020 • edited Loading

thekevinbrown commented Feb 20, 2020

thekevinbrown commented Feb 19, 2020 •

edited

Loading

jogold commented Feb 19, 2020 •

edited

Loading

thekevinbrown commented Feb 19, 2020 •

edited

Loading

thekevinbrown commented Feb 19, 2020 •

edited

Loading

thekevinbrown commented Feb 19, 2020 •

edited

Loading

jogold commented Feb 19, 2020 •

edited

Loading