(aws-secretsmanager): grant read fails when secret is fetched using fromSecretNamev2

[mneirynck]

What is the problem?

I have manually created a secret named "SECRET-NAME" in SecretsManager and need to request it in one of my stacks, I'm doing it like so:

secret = SMSecret.from_secret_name_v2(
            self, "Secret", "SECRET-NAME"
        )

When I grant read access to a role like so:

secret.grant_read(role)

It generates the wrong access:

            [ ] {
            [ ]   "Action": [
            [+]     "secretsmanager:GetSecretValue",
            [+]     "secretsmanager:DescribeSecret"
            [+]   ],
            [+]   "Effect": "Allow",
            [+]   "Resource": {
            [+]     "Fn::Join": [
            [+]       "",
            [+]       [
            [+]         "arn:",
            [+]         {
            [+]           "Ref": "AWS::Partition"
            [+]         },
            [+]         ":secretsmanager:eu-central-1:726654634199:secret:SECRET-NAME-??????"
            [+]       ]
            [+]     ]
            [+]   }
            [+] },

Reproduction Steps

Create a manual secret without the secretsmanager added suffix and request it from within your code.

What did you expect to happen?

I would have expected my role to have read access to the role

What actually happened?

A non-existing ARN was used as the secret ARN in the policy

CDK CLI Version

2.0.0-rc.33

Framework Version

No response

Node.js Version

14.16.0

OS

Arch Linux

Language

Python

Language Version

Python 3.9.9

Other information

No response

[njlynch]

Hi @mneirynck ,

Can you provide some more detail? Does the role actually receive an error when trying to fetch the secret?

A non-existing ARN was used as the secret ARN in the policy

Can you clarify this? The ARN as shown in your issue description looks correct to me. It's worth noting that the ? characters in the ARN are wildcards, and are used to represent the 6-character suffix SecretsManager automatically adds to any secret name.

For example, if you create a secret with a name of mySecret, the ARN might end up like: arn:secretsmanager:eu-central-1:123456789012:secret:mySecret-da1c22. The ARN generated by the CDK for the policy would be arn:secretsmanager:eu-central-1:123456789012:secret:mySecret-??????, which would match.

Given your reproduction steps, the role I've created can access the secret perfectly fine. Can you clarify, and/or provide updated reproduction steps?

[github-actions]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[CarsonF]

I'm experiencing this as well.
Edit: Nevermind my problem was an imported secret did exist. I wish CDK warned me about that..

Original message

The IAM role grants

arn:aws:secretsmanager:{region}:{account}:secret:{name}-??????

ECS task asks for

arn:aws:secretsmanager:{region}:{account}:secret:{name}:{field}::

Simulating the IAM role policy, grants fail for all wildcards:

Implicitly denied (no matching statements).

Only grants when asking for specific {name}-?????? suffix.
These all are denied:

arn:aws:secretsmanager:{region}:{account}:secret:{name}
arn:aws:secretsmanager:{region}:{account}:secret:{name}:{field}::
arn:aws:secretsmanager:{region}:{account}:secret:{name}:::

[mneirynck]

It's probably not a problem with SecretsManager after all, but the way an ECS task requests the secret.
The CDK deployment runs without issues, but the container can't start because it can't find the secret.

from aws_cdk.aws_secretsmanager import Secret as SMSecret
from aws_cdk.aws_ecs import Secret
from aws_cdk.aws_ecs_pattersn import ApplicationLoadBalancedFargateService, ApplicationLoadBalancedTaskImageOptions

secret = SMSecret.from_secret_name_v2(self, "Secret", "SECRET-NAME")
task_secrets = {
    "SECRET_USER": Secret.from_secrets_manager(secret,"username"),
    "SECRET_PASSWORD": Secret.from_secrets_manager(secret,"password")
    }

service = ApplicationLoadBalancedFargateService(
    self,
    "Service",
    cluster = ecs_cluster,
    task_image_options = ApplicationLoadBalancedTaskImageOptions(
        container_name="App",
        image = image,
        secrets = task_secrets
    ),
    task_subnets=SubnetSelection(subnets=vpc.private_subnets),
)

task_role = service.task_definition.task_role

secret.grant_read(task_role)

That's sort of the code I'm using in the stack. The container fails to start with the following message:

I've tried both task_role and execution_role in the grant_read, but both give the same result.

[CarsonF]

@mneirynck there's two roles here. The task role is what the runtime docker code uses (your app code). The execution role is what Fargate uses to fetch the docker image, fetch aws secrets, etc in order to setup the containers.
CDK automatically grants read access for the secrets given to execution role, so your grant here isn't needed.

I'd look in CloudTrail. I had the exact same error message and I finally found the answer there. My secret I was referencing actually didn't exist.
You can search in the events. Filter by username and use the task ID as the user name. I eventually found this:

    "errorCode": "ResourceNotFoundException",
    "errorMessage": "Secrets Manager can't find the specified secret.",
    "requestParameters": {
        "secretId": "arn:aws:secretsmanager:us-east-2:{account}:secret:{name}"
    },

[mneirynck]

Hi,
Thanks for letting me check CloudTrail, seems like there's no policy created for accessing the secret:

"errorCode": "AccessDenied",
    "errorMessage": "User: arn:aws:sts::{account}:assumed-role/{Service}ServiceTaskDefExecutionRole is not authorized to perform: secretsmanager:GetSecretValue on resource: arn:aws:secretsmanager:eu-central-1:{account}:secret:{name} because no identity-based policy allows the secretsmanager:GetSecretValue action",

Is it by design we need to manually add it to the FargateService? Before it was done automatically I believe since I have services where I'm doing the same thing actually.

EDIT: it is getting added by cdk, so that's not the problem either

[mneirynck]

Nevermind, I'm just plain stupid...

Since it's a manually created secret I have to add the Policy manually to the secret as well (or via cdk explicitly)

EDIT:
On the other hand, I've just checked some other CDK created secrets and there's no policy over there either. So the issue remains.

Cloudtrail event:

{
    "eventVersion": "1.08",
    "userIdentity": {
        "type": "AssumedRole",
        "principalId": "{principal}:6585fdab53ab4591b09c54c6bd23140b",
        "arn": "arn:aws:sts::{account}:assumed-role/SERVICE-LAB-ServiceTaskDefExecutionRole919F7BE3-14SNWF2X5VJTW/6585fdab53ab4591b09c54c6bd23140b",
        "accountId": "{account}",
        "accessKeyId": "{access_key}",
        "sessionContext": {
            "sessionIssuer": {
                "type": "Role",
                "principalId": "{principal}",
                "arn": "arn:aws:iam::{account}:role/SERVICE-LAB-ServiceTaskDefExecutionRole919F7BE3-14SNWF2X5VJTW",
                "accountId": "{account}",
                "userName": "SERVICE-LAB-ServiceTaskDefExecutionRole919F7BE3-14SNWF2X5VJTW"
            },
            "webIdFederationData": {},
            "attributes": {
                "creationDate": "2022-01-04T19:51:35Z",
                "mfaAuthenticated": "false"
            }
        }
    },
    "eventTime": "2022-01-04T19:51:36Z",
    "eventSource": "secretsmanager.amazonaws.com",
    "eventName": "GetSecretValue",
    "awsRegion": "eu-central-1",
    "sourceIPAddress": "18.198.191.233",
    "userAgent": "Amazon Fargate Agent - v (*) (linux) (+http://aws.amazon.com/ecs/)",
    "errorCode": "AccessDenied",
    "errorMessage": "User: arn:aws:sts::{account}:assumed-role/SERVICE-LAB-ServiceTaskDefExecutionRole is not authorized to perform: secretsmanager:GetSecretValue on resource: arn:aws:secretsmanager:eu-central-1:{account}:secret:SECRET-NAME because no identity-based policy allows the secretsmanager:GetSecretValue action",
    "requestParameters": null,
    "responseElements": null,
    "requestID": "d9bb5f65-3304-491b-bd8e-10ca7a03eac7",
    "eventID": "635a2be7-0c0c-4b77-9dfe-bbb2bb64aaaf",
    "readOnly": true,
    "eventType": "AwsApiCall",
    "managementEvent": true,
    "recipientAccountId": "{account}",
    "eventCategory": "Management",
    "tlsDetails": {
        "clientProvidedHostHeader": "secretsmanager.eu-central-1.amazonaws.com"
    }
}

Everything works if I use the secret.from_arn thing and fill in the arn of the secret, so there is something different between the two. For good measure, this is the ARN of the secret:
arn:aws:secretsmanager:eu-central-1:{account}:secret:SECRET-NAME-T23Sm2

So the only thing I see differently is the ErrorMessage states an ARN without the suffix, but the policy has the suffix. But since it's not a secret not found error I assume it finds the correct secret.

[mneirynck]

I'll try with another secret name, as I was indeed using SOMETHING-SECRET as the secret name, which happens to be 6 chars after the hyphen.

[peterwoodworth]

I believe this PR is set to fix this issue.

[mneirynck]

Nope, still doesn't work on 2.10.0. I still need to try with a different secret name, but I guess that will work since all our other secrets don't have those 6 chars in the name.

[CarsonF]

Ok I'm hitting this again too.

User: arn:aws:sts::{}:assumed-role/seed-dev-APIFargateTaskDefExecutionRole00F98ADA-112C4U0TJQ47V/cfabb1745ae44a5bbd5aaca3e04359b3 is not authorized to perform: secretsmanager:GetSecretValue on resource: arn:aws:secretsmanager:us-east-2:{}:secret:seed-api/apollo-studio because no identity-based policy allows the secretsmanager:GetSecretValue action

In the policy referenced I have it...

        {
            "Action": [
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret"
            ],
            "Resource": "arn:aws:secretsmanager:us-east-2:{}:secret:plaid/sandbox-??????",
            "Effect": "Allow"
        },
        {
            "Action": [
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret"
            ],
            "Resource": "arn:aws:secretsmanager:us-east-2:{}:secret:seed-api/apollo-studio-??????",
            "Effect": "Allow"
        }

Above it is another secret that works fine.

Surely it couldn't be because the secret ends in a 6 char word....

[CarsonF]

Well I changed the secret name from seed-api/apollo-studio to seed-api/apollo-studio-key and it worked. Wow I think that might be it.

[peterwoodworth]

I'll try to reproduce this next week at some point. Thanks for the find @CarsonF

[mneirynck]

I just tested it with a different name and it worked as long as it doesn't end with 6 characters.

[sammcj]

We just encountered this issue with aws-cdk 2.20.0 and it was the 6 character problem 🤯

ms-shared-secret

fails

ms-shared-key

works

[chefren]

IAM doc is not too clear on permission for secrets, it shows an arn example with {SecretID} at the end, but the reality is secrets have an ARN with a 6 char hash at the end.

It is a bit clearer on the Secrets manager permissions doc, maybe the policy is generated by CDK and doesn't contemplate this particularity.

Secrets Manager constructs the last part of the secret ARN by appending a dash and six random alphanumeric characters at the end of the secret name. If you delete a secret and then recreate another with the same name, this formatting helps ensure that individuals with permissions to the original secret don't automatically get access to the new secret because Secrets Manager generates six new random characters.

You can find the ARN for a secret in the Secrets Manager console on the secret details page or by calling [DescribeSecret](https://docs.aws.amazon.com/secretsmanager/latest/apireference/API_DescribeSecret.html).

[chefren]

Well I changed the secret name from seed-api/apollo-studio to seed-api/apollo-studio-key and it worked. Wow I think that might be it.

@CarsonF note that the generated permission ended in apollo-studio-??????, so seed-api/apollo-studio wouldn't match as it doesn't end with "-"

[chefren]

I believe this PR is set to fix this issue.

That definitely relates to the problem. I think the root cause is any time the secret is required, should attempt to access via full ARN instead of by name due to the IAM policy specification.

[chefren]

Here's the doc that explains it

[github-actions]

This issue has not received any attention in 1 year. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[chefren]

Happy birthday issue :)

[peterwoodworth]

Given the documentation linked above, and based on my previous investigation into this and other related issues, I don't think there's anything CDK can do here. We specifically call out in our docs to use fromSecretCompleteArn() when your secret name ends in a hyphen and 6 characters due to this service limitation.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[marcesengel]

@peterwoodworth would it be reasonable to print a warning or even throw an exception under the given circumstances, either on import or when trying to use the secret imported this way with the ECS secrets feature? I even had this issue before and still struggled for a bit today after making the foolish mistake of ending my secret name with -secret 😅

[peterwoodworth]

@marcesengel I'd recommend opening up another feature request or pinging another team member, I no longer work with this product 🙂