Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

iam_role is ignored for dependencies #1262

Open
Spaideri opened this issue Nov 21, 2022 · 11 comments
Open

iam_role is ignored for dependencies #1262

Spaideri opened this issue Nov 21, 2022 · 11 comments

Comments

@Spaideri
Copy link

iam_role resolver is ignored for the stack refrenced with !stack_output resolver

  • Stack A and stack B have both iam_role defined.
  • Stack A has a dependency to stack B through a !stack_output reference
  • Launching stack A causes naturally the depending stack B to be launched first, but instead of assuming the iam_role defined for stack B the it is launched using the current credentials, after stack B launch is complete stack A is executed with the specified iam_role for the stack A as expected
  • In my example case I've defined the same iam_role for A and B and my current user is not authorized to assume that role, but I'm still able to launch the transient stack B by launching stack A

Your environment

  • version of sceptre (sceptre --version): 3.2.0
  • version of python (python --version): 3.9.9
  • which OS/distro: macOS 12.6

Steps to reproduce

Note current user is not allowed to assume the IacInfraDeployRoleArn role

stack B

template_path: route53/cf-route53-private-hosted-zone.yaml

iam_role: !stack_output_external managed-default-iac-roles::IacInfraDeployRoleArn

stack A

template_path: iam/cf-iam-infra-roles.yaml

iam_role: !stack_output_external managed-default-iac-roles::IacInfraDeployRoleArn

parameters:
  PrivateHostedZoneId: !stack_output {{ stack_group_config.environment }}/eu-west-1/infra/route53-private-hosted-zone.yaml::HostedZoneID

Output launching stack A

sceptre launch dev/eu-west-1/infra/iam-infra-roles.yaml -y
[2022-11-21 13:28:18] - dev/eu-west-1/infra/route53-private-hosted-zone - Launching Stack
[2022-11-21 13:28:19] - dev/eu-west-1/infra/route53-private-hosted-zone - Stack is in the UPDATE_COMPLETE state
[2022-11-21 13:28:19] - dev/eu-west-1/infra/route53-private-hosted-zone - Updating Stack
[2022-11-21 13:28:20] - dev/eu-west-1/infra/route53-private-hosted-zone projectcode-dev-eu-west-1-infra-route53-private-hosted-zone AWS::CloudFormation::Stack UPDATE_IN_PROGRESS User Initiated
[2022-11-21 13:28:24] - dev/eu-west-1/infra/route53-private-hosted-zone PrivateHostedZone AWS::Route53::HostedZone UPDATE_IN_PROGRESS
[2022-11-21 13:28:50] - dev/eu-west-1/infra/route53-private-hosted-zone PrivateHostedZone AWS::Route53::HostedZone UPDATE_COMPLETE
[2022-11-21 13:28:50] - dev/eu-west-1/infra/route53-private-hosted-zone projectcode-dev-eu-west-1-infra-route53-private-hosted-zone AWS::CloudFormation::Stack UPDATE_COMPLETE_CLEANUP_IN_PROGRESS
[2022-11-21 13:28:50] - dev/eu-west-1/infra/route53-private-hosted-zone projectcode-dev-eu-west-1-infra-route53-private-hosted-zone AWS::CloudFormation::Stack UPDATE_COMPLETE
"An error occurred (AccessDenied) when calling the AssumeRole operation: User: arn:aws:sts::1234567890:assumed-role/AWSReservedSSO_DeveloperAccess_xyz/user@domain.com is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::1234567890:role/iac-roles/infra/projectcode-dev-iac-infra-deploy-role"

Output launching stack B directly

sceptre launch dev/eu-west-1/infra/route53-private-hosted-zone.yaml -y
[2022-11-21 13:29:07] - dev/eu-west-1/infra/route53-private-hosted-zone - Launching Stack
"An error occurred (AccessDenied) when calling the AssumeRole operation: User: arn:aws:sts::1234567890:assumed-role/AWSReservedSSO_DeveloperAccess_xyz/user@domain.com is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::1234567890:role/iac-roles/infra/projectcode-dev-iac-infra-deploy-role"

Expected behaviour

Stack dependencies defined throug the !stack_output resolver should be executed with the iam_role defined for the stack

Actual behaviour

Dependency stacks through !stack_output are executed with the current users credentials instead of assuming the iam_role defined for the dependency stack

@jfalkenstein
Copy link
Contributor

Hi @Spaideri, there are a few things going on here, so let me comment on them individually.

  1. Whenever you use iam_role as an argument, you're telling Sceptre to role-chain (see snippet from AWS docs below). This means that Sceptre will FIRST use the credentials of the deploying environment (via the profile, aws credentials file, and/or environment variables). Then it will assume the iam_role using the permissions of your environment's role and use that iam_role to perform further actions.
  2. When !stack_output is used, the iam_role, profile, and region of the TARGETED stack is used to obtain the outputs, not the stack getting the outputs.
  3. There is a caveat to the above Add HTTPS support for documentation #1 and Load only external hook classes required by relevant stack config(s) #2. If you're setting iam_role with !stack_output or !stack_output_external, if it hasn't yet resolved the iam_role arn of the target stack and it can't because of a recursive resolve, it will fall back using the environment's credentials. This caveat is carved out in code here and is how we were able to support having a resolvable iam_role. This is required in order to allow iam_role to be resolvable in the first place. I suspect this area of code is currently the source of the bug you've discovered.

I was the one that originally wrote that piece of code. The goal was to support using stack_output and other resolvers for iam_role. However, as I look at it, I suspect what is happening during your deployment is that there is another source of "recursive resolve" in your stack configurations which is triggering the fallback to environment credentials.

Correct me if I'm wrong, but based upon what you've posted, I'd expect the expected behavior in your case to be NOTHING deploys because your user is unable to assume the iam role.

What I'm currently thinking the fix should be: Right now, the fallback happens when the ConnectionManager is obtained. However, what I'm thinking we should instead is tool out that fallback in ONLY the !stack_output and !stack_output_external resolvers. (If you're resolving and iam_role, you need to use environment permissions to at least get that stack_output first before you can assume it).

image

@jfalkenstein
Copy link
Contributor

@Spaideri, thanks so much for reporting this bug. It's a bit of an obscure edge-case and I'm still not positive what is actually causing this other-case fallback in your stack configuration, but I think it needs to be fixed.

First, in order to prove my hypothesis, could you do me a favor and try out deploying but replace the iam_role value with the actual hard-coded role ARN rather than using the resolver. If I'm correct, no stack should be able to be deployed and it should fail fast. Please let me know the results.

Second, how would you like to contribute toward a fix? I'd be happy to help guide you/coach you in the process. There's a good community of Sceptre devs and I've found we're quite welcoming! (I only started up about a year and a half ago).

@Spaideri
Copy link
Author

Hi, and thanks for prompt reply!

You are right to expect that nothing should be deployed if the specified iam_role can't be assumed with the active credentials

I tested with hard coded iam_role values and the behaviour is the same

sceptre launch dev/eu-west-1/infra/iam-infra-roles.yaml -y
[2022-11-22 09:25:04] - dev/eu-west-1/infra/route53-env-public-hz - Launching Stack
[2022-11-22 09:25:04] - dev/eu-west-1/infra/route53-private-hosted-zone - Launching Stack
[2022-11-22 09:25:05] - dev/eu-west-1/infra/route53-private-hosted-zone - Stack is in the UPDATE_COMPLETE state
[2022-11-22 09:25:05] - dev/eu-west-1/infra/route53-private-hosted-zone - Updating Stack
[2022-11-22 09:25:05] - dev/eu-west-1/infra/route53-private-hosted-zone - No updates to perform.
"An error occurred (AccessDenied) when calling the AssumeRole operation: User: arn:aws:sts::1234567890:assumed-role/AWSReservedSSO_DeveloperAccess_xyz/user@domain.com is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::1234567890:role/iac-roles/infra/projectcode-dev-iac-infra-deploy-role"

Full config files below. I've simplified them a bit by removing some jinja2 imports and custom resolvers, but these are the actual versions I've reproduced the error with

route53-private-hosted-zone.yaml "Stack B"

template_path: route53/cf-route53-private-hosted-zone.yaml

iam_role: arn:aws:iam::1234567890:role/iac-roles/infra/project-dev-iac-infra-deploy-role

stack_tags:
  environment: {{ stack_group_config.environment }}
  application-id : '{{ stack_group_config.tags.application_id }}'
  application-name : {{ stack_group_config.tags.project_name }}
  cost-center : '{{ stack_group_config.tags.cost_center }}'

parameters:
  VpcId: vpc-123xyz
  ApplicationName: {{ stack_group_config.project_code }}
  Environment: {{ stack_group_config.environment }}

iam-infra-roles.yaml "Stack A"

template_path: iam/cf-iam-infra-roles.yaml

iam_role: arn:aws:iam::1234567890:role/iac-roles/infra/project-dev-iac-infra-deploy-role

stack_tags:
  environment: {{ stack_group_config.environment }}
  application-id : '{{ stack_group_config.tags.application_id }}'
  application-name : {{ stack_group_config.tags.project_name }}
  cost-center : '{{ stack_group_config.tags.cost_center }}'

parameters:
  ApplicationName: {{ stack_group_config.project_code }}
  ApplicationId: '{{ stack_group_config.tags.application_id }}'
  Environment: {{ stack_group_config.environment }}
  KmsEnvironmentKeyArns:
    - arn:aws:kms:eu-west-1:1234567890:key/*
  PublicHostedZoneId: !stack_output {{ stack_group_config.environment }}/eu-west-1/infra/route53-env-public-hz.yaml::HostedZoneID
  PrivateHostedZoneId: !stack_output {{ stack_group_config.environment }}/eu-west-1/infra/route53-private-hosted-zone.yaml::HostedZoneID
  PermissionBoundaryArn: !stack_output_external project-{{ stack_group_config.environment }}-{{ stack_group_config.tags.project_name }}-eu-west-1-default-boundary-policy::PermissionBoundaryPolicyArn

Regarding contributing, python is the weakest language in my tool box and I feel like I'm not able to produce added value at this time :(

@jfalkenstein
Copy link
Contributor

I think it actually worked when you hard-coded. I believe AWS doesn't bother checking permissions when you attempt to deploy a stack with no changes. It Just says "no changes to be made" and doesn't blow up. That appears to be what actually happened. If I'm right, you it should block any changes to the stack. If you modify the stack template for your hosted zone and try to deploy, it SHOULD blow up without making any changes. (Do something non-destructive in case it's your prod thing. Like add a character to the template "Description" or something.

Regarding contributing, python is the weakest language in my tool box and I feel like I'm not able to produce added value at this time :(

That's fine. I'll see what I can do when I get bored over the coming holiday ; ).

@Spaideri
Copy link
Author

Hi, I tried again with the hard coded iam_role and I can confirm that the problem occurs with the hard coded values

Launch with hard coded iam role, I updated private hosted zone stack tags to trigger changes

sceptre launch dev/eu-west-1/infra/iam-infra-roles.yaml -y
[2022-11-23 09:41:01] - dev/eu-west-1/infra/route53-private-hosted-zone - Launching Stack
[2022-11-23 09:41:02] - dev/eu-west-1/infra/route53-private-hosted-zone - Stack is in the UPDATE_COMPLETE state
[2022-11-23 09:41:02] - dev/eu-west-1/infra/route53-private-hosted-zone - Updating Stack
[2022-11-23 09:41:03] - dev/eu-west-1/infra/route53-private-hosted-zone project-dev-eu-west-1-infra-route53-private-hosted-zone AWS::CloudFormation::Stack UPDATE_IN_PROGRESS User Initiated
[2022-11-23 09:41:07] - dev/eu-west-1/infra/route53-private-hosted-zone PrivateHostedZone AWS::Route53::HostedZone UPDATE_IN_PROGRESS
[2022-11-23 09:41:33] - dev/eu-west-1/infra/route53-private-hosted-zone PrivateHostedZone AWS::Route53::HostedZone UPDATE_COMPLETE
[2022-11-23 09:41:33] - dev/eu-west-1/infra/route53-private-hosted-zone project-dev-eu-west-1-infra-route53-private-hosted-zone AWS::CloudFormation::Stack UPDATE_COMPLETE_CLEANUP_IN_PROGRESS
[2022-11-23 09:41:33] - dev/eu-west-1/infra/route53-private-hosted-zone project-dev-eu-west-1-infra-route53-private-hosted-zone AWS::CloudFormation::Stack UPDATE_COMPLETE
"An error occurred (AccessDenied) when calling the AssumeRole operation: User: arn:aws:sts::1234567890:assumed-role/AWSReservedSSO_DeveloperAccess_123xyz/user@domain.com is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::1234567890:role/iac-roles/infra/project-dev-iac-infra-deploy-role"

Launching the private hosted zone stack directly

sceptre launch dev/eu-west-1/infra/route53-private-hosted-zone.yaml -y
[2022-11-23 09:42:24] - dev/eu-west-1/infra/route53-private-hosted-zone - Launching Stack
"An error occurred (AccessDenied) when calling the AssumeRole operation: User: arn:aws:sts::1234567890:assumed-role/AWSReservedSSO_DeveloperAccess_123xyz/user@domain.com is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::1234567890:role/iac-roles/infra/project-dev-iac-infra-deploy-role"

Hope you have a very very relaxing holiday! :)

@jfalkenstein
Copy link
Contributor

Huh. Well that isn't the result is was hoping for. I'll need to investigate. I still think there's some sort of catching going on.

@jfalkenstein
Copy link
Contributor

Hey @Spaideri, using the hard-coded iam_roles (not resolvers for them), can you run the launch using the --debug flag and post the full logs. There are a bunch of debug logs that happen around IAM role assumption and I want to see what's going on.

@Spaideri
Copy link
Author

Hi!

Weird thing is that running with --debug the execution fails before the depending stack is launched. I removed some noise from the beginning of output

sceptre --debug launch dev/eu-west-1/infra/iam-infra-roles.yaml -y

[2022-11-24 16:22:20] - Generate dependencies for stack dev/eu-west-1/infra/route53-private-hosted-zone
[2022-11-24 16:22:20] - Generate dependencies for stack dev/eu-west-1/infra/kms-environment-key
[2022-11-24 16:22:20] - Generate dependencies for stack dev/eu-west-1/infra/route53-env-public-hz
[2022-11-24 16:22:20] - Generate dependencies for stack dev/eu-west-1/infra/iam-infra-roles
[2022-11-24 16:22:20] -   Added dependency: dev/eu-west-1/infra/route53-private-hosted-zone
[2022-11-24 16:22:20] -   Added dependency: dev/eu-west-1/infra/route53-env-public-hz
[2022-11-24 16:22:20] -   Added dependency: dev/eu-west-1/infra/kms-environment-key
[2022-11-24 16:22:20] - dev/eu-west-1/infra/route53-private-hosted-zone - Launching Stack
[2022-11-24 16:22:20] - No cloudformation client found, creating one...
[2022-11-24 16:22:20] - Resolving external Stack output: project-cloud-access-eu-west-1-app-multiaccounts-projecttemplate-dev-automate-managed-default-iac-roles::IacInfraDeployRoleArn
[2022-11-24 16:22:20] - Getting Boto3 session
[2022-11-24 16:22:20] - Collecting outputs from 'project-cloud-access-eu-west-1-app-multiaccounts-projecttemplate-dev-automate-managed-default-iac-roles'...
[2022-11-24 16:22:20] - No Boto3 session found, creating one...
[2022-11-24 16:22:20] - Resolving iam_role requires the Stack connection manager. Temporarily setting the iam_role to None until it can be fully resolved.
[2022-11-24 16:22:20] - Using cli credentials...
[2022-11-24 16:22:22] - No cloudformation client found, creating one...
[2022-11-24 16:22:22] - Getting Boto3 session
[2022-11-24 16:22:22] - No Boto3 session found, creating one...
[2022-11-24 16:22:22] - Using cli credentials...
[2022-11-24 16:22:22] - Using credential set from sso: {'AccessKeyId': '****************XYZ', 'SecretAccessKey': '************************************XYZ', 'Region': 'eu-west-1'}
[2022-11-24 16:22:22] - Boto3 session created
[2022-11-24 16:22:22] - Outputs: [{'OutputKey': 'IacInfraDeployRoleArn', 'OutputValue': 'arn:aws:iam::1234567890:role/iac-roles/infra/projecttemplate-dev-iac-infra-deploy-role', 'ExportName': 'project-cloud-access-eu-west-1-app-multiaccounts-projecttemplate-dev-automate-managed-default-iac-roles:IacInfraDeployRoleArn'}, {'OutputKey': 'IacAppDeployRoleArn', 'OutputValue': 'arn:aws:iam::1234567890:role/iac-roles/app/projecttemplate-dev-iac-app-deploy-role', 'ExportName': 'project-cloud-access-eu-west-1-app-multiaccounts-projecttemplate-dev-automate-managed-default-iac-roles:IacAppDeployRoleArn'}, {'OutputKey': 'IacInfraDeployRoleName', 'OutputValue': 'projecttemplate-dev-iac-infra-deploy-role', 'ExportName': 'project-cloud-access-eu-west-1-app-multiaccounts-projecttemplate-dev-automate-managed-default-iac-roles:IacInfraDeployRoleName'}, {'OutputKey': 'IacAppDeployRoleName', 'OutputValue': 'projecttemplate-dev-iac-app-deploy-role', 'ExportName': 'project-cloud-access-eu-west-1-app-multiaccounts-projecttemplate-dev-automate-managed-default-iac-roles:IacAppDeployRoleName'}]
[2022-11-24 16:22:22] - dev/eu-west-1/infra/route53-env-public-hz - Launching Stack
[2022-11-24 16:22:22] - No cloudformation client found, creating one...
[2022-11-24 16:22:22] - Resolving external Stack output: project-cloud-access-eu-west-1-app-multiaccounts-projecttemplate-dev-automate-managed-default-iac-roles::IacInfraDeployRoleArn
[2022-11-24 16:22:22] - Getting Boto3 session
[2022-11-24 16:22:22] - Collecting outputs from 'project-cloud-access-eu-west-1-app-multiaccounts-projecttemplate-dev-automate-managed-default-iac-roles'...
[2022-11-24 16:22:22] - Resolving iam_role requires the Stack connection manager. Temporarily setting the iam_role to None until it can be fully resolved.
[2022-11-24 16:22:22] - Outputs: [{'OutputKey': 'IacInfraDeployRoleArn', 'OutputValue': 'arn:aws:iam::1234567890:role/iac-roles/infra/projecttemplate-dev-iac-infra-deploy-role', 'ExportName': 'project-cloud-access-eu-west-1-app-multiaccounts-projecttemplate-dev-automate-managed-default-iac-roles:IacInfraDeployRoleArn'}, {'OutputKey': 'IacAppDeployRoleArn', 'OutputValue': 'arn:aws:iam::1234567890:role/iac-roles/app/projecttemplate-dev-iac-app-deploy-role', 'ExportName': 'project-cloud-access-eu-west-1-app-multiaccounts-projecttemplate-dev-automate-managed-default-iac-roles:IacAppDeployRoleArn'}, {'OutputKey': 'IacInfraDeployRoleName', 'OutputValue': 'projecttemplate-dev-iac-infra-deploy-role', 'ExportName': 'project-cloud-access-eu-west-1-app-multiaccounts-projecttemplate-dev-automate-managed-default-iac-roles:IacInfraDeployRoleName'}, {'OutputKey': 'IacAppDeployRoleName', 'OutputValue': 'projecttemplate-dev-iac-app-deploy-role', 'ExportName': 'project-cloud-access-eu-west-1-app-multiaccounts-projecttemplate-dev-automate-managed-default-iac-roles:IacAppDeployRoleName'}]
[2022-11-24 16:22:22] - dev/eu-west-1/infra/kms-environment-key - Launching Stack
[2022-11-24 16:22:23] - dev/eu-west-1/infra/route53-env-public-hz - Stack is in the UPDATE_COMPLETE state
[2022-11-24 16:22:23] - dev/eu-west-1/infra/route53-env-public-hz - Updating Stack
[2022-11-24 16:22:23] - dev/eu-west-1/infra/kms-environment-key - Stack is in the UPDATE_COMPLETE state
[2022-11-24 16:22:23] - dev/eu-west-1/infra/kms-environment-key - Updating Stack
[2022-11-24 16:22:23] - dev/eu-west-1/infra/route53-env-public-hz - No updates to perform.
[2022-11-24 16:22:23] - dev/eu-west-1/infra/kms-environment-key - No updates to perform.
[2022-11-24 16:22:23] - No cloudformation client found, creating one...
[2022-11-24 16:22:23] - Getting Boto3 session
[2022-11-24 16:22:23] - No cloudformation client found, creating one...
[2022-11-24 16:22:23] - Getting Boto3 session
[2022-11-24 16:22:24] - dev/eu-west-1/infra/route53-env-public-hz - termination protection set to 'enabled'
[2022-11-24 16:22:24] - dev/eu-west-1/infra/kms-environment-key - termination protection set to 'enabled'
Traceback (most recent call last):
  File "/Users/username/ws/xxxx/project/projecttemplate-cloud-infra/venv/bin/sceptre", line 8, in <module>
    sys.exit(cli())
  File "/Users/username/ws/xxxx/project/projecttemplate-cloud-infra/venv/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/username/ws/xxxx/project/projecttemplate-cloud-infra/venv/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/username/ws/xxxx/project/projecttemplate-cloud-infra/venv/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/username/ws/xxxx/project/projecttemplate-cloud-infra/venv/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/username/ws/xxxx/project/projecttemplate-cloud-infra/venv/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/username/ws/xxxx/project/projecttemplate-cloud-infra/venv/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/username/ws/xxxx/project/projecttemplate-cloud-infra/venv/lib/python3.9/site-packages/sceptre/cli/helpers.py", line 42, in decorated
    return func(*args, **kwargs)
  File "/Users/username/ws/xxxx/project/projecttemplate-cloud-infra/venv/lib/python3.9/site-packages/sceptre/cli/launch.py", line 54, in launch_command
    exit_code = launcher.launch(prune)
  File "/Users/username/ws/xxxx/project/projecttemplate-cloud-infra/venv/lib/python3.9/site-packages/sceptre/cli/launch.py", line 94, in launch
    code = code or self._deploy(deploy_plan)
  File "/Users/username/ws/xxxx/project/projecttemplate-cloud-infra/venv/lib/python3.9/site-packages/sceptre/cli/launch.py", line 178, in _deploy
    result = deploy_plan.launch()
  File "/Users/username/ws/xxxx/project/projecttemplate-cloud-infra/venv/lib/python3.9/site-packages/sceptre/plan/plan.py", line 169, in launch
    return self._execute(*args)
  File "/Users/username/ws/xxxx/project/projecttemplate-cloud-infra/venv/lib/python3.9/site-packages/sceptre/plan/plan.py", line 29, in wrapped
    return func(self, *args, **kwargs)
  File "/Users/username/ws/xxxx/project/projecttemplate-cloud-infra/venv/lib/python3.9/site-packages/sceptre/plan/plan.py", line 56, in _execute
    return executor.execute(*args)
  File "/Users/username/ws/xxxx/project/projecttemplate-cloud-infra/venv/lib/python3.9/site-packages/sceptre/plan/executor.py", line 52, in execute
    stack, status = future.result()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 438, in result
    return self.__get_result()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 390, in __get_result
    raise self._exception
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/username/ws/xxxx/project/projecttemplate-cloud-infra/venv/lib/python3.9/site-packages/sceptre/plan/executor.py", line 59, in _execute
    result = getattr(actions, self.command)(*args)
  File "/Users/username/ws/xxxx/project/projecttemplate-cloud-infra/venv/lib/python3.9/site-packages/sceptre/hooks/__init__.py", line 104, in decorated
    response = func(self, *args, **kwargs)
  File "/Users/username/ws/xxxx/project/projecttemplate-cloud-infra/venv/lib/python3.9/site-packages/sceptre/plan/actions.py", line 201, in launch
    existing_status = self._get_status()
  File "/Users/username/ws/xxxx/project/projecttemplate-cloud-infra/venv/lib/python3.9/site-packages/sceptre/plan/actions.py", line 813, in _get_status
    raise exp
  File "/Users/username/ws/xxxx/project/projecttemplate-cloud-infra/venv/lib/python3.9/site-packages/sceptre/plan/actions.py", line 808, in _get_status
    status = self._describe()["Stacks"][0]["StackStatus"]
  File "/Users/username/ws/xxxx/project/projecttemplate-cloud-infra/venv/lib/python3.9/site-packages/sceptre/plan/actions.py", line 800, in _describe
    return self.connection_manager.call(
  File "/Users/username/ws/xxxx/project/projecttemplate-cloud-infra/venv/lib/python3.9/site-packages/sceptre/connection_manager.py", line 51, in decorated
    return func(*args, **kwargs)
  File "/Users/username/ws/xxxx/project/projecttemplate-cloud-infra/venv/lib/python3.9/site-packages/sceptre/connection_manager.py", line 260, in call
    client = self._get_client(service, region, profile, stack_name, iam_role)
  File "/Users/username/ws/xxxx/project/projecttemplate-cloud-infra/venv/lib/python3.9/site-packages/sceptre/connection_manager.py", line 224, in _get_client
    self._clients[key] = self._get_session(
  File "/Users/username/ws/xxxx/project/projecttemplate-cloud-infra/venv/lib/python3.9/site-packages/sceptre/connection_manager.py", line 169, in _get_session
    sts_response = sts_client.assume_role(**assume_role_kwargs)
  File "/Users/username/ws/xxxx/project/projecttemplate-cloud-infra/venv/lib/python3.9/site-packages/botocore/client.py", line 514, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/username/ws/xxxx/project/projecttemplate-cloud-infra/venv/lib/python3.9/site-packages/botocore/client.py", line 938, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the AssumeRole operation: User: arn:aws:sts::1234567890:assumed-role/AWSReservedSSO_DeveloperAccess_xyz/ext-username.yyy@xxxx.com is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::1234567890:role/iac-roles/infra/projecttemplate-dev-iac-infra-deploy-role

Without --debug the result is as before

sceptre launch dev/eu-west-1/infra/iam-infra-roles.yaml -y

[2022-11-24 16:28:38] - dev/eu-west-1/infra/route53-env-public-hz - Launching Stack
[2022-11-24 16:28:38] - dev/eu-west-1/infra/route53-private-hosted-zone - Launching Stack
[2022-11-24 16:28:39] - dev/eu-west-1/infra/kms-environment-key - Launching Stack
[2022-11-24 16:28:39] - dev/eu-west-1/infra/route53-private-hosted-zone - Stack is in the UPDATE_COMPLETE state
[2022-11-24 16:28:39] - dev/eu-west-1/infra/route53-private-hosted-zone - Updating Stack
[2022-11-24 16:28:39] - dev/eu-west-1/infra/kms-environment-key - Stack is in the UPDATE_COMPLETE state
[2022-11-24 16:28:39] - dev/eu-west-1/infra/kms-environment-key - Updating Stack
[2022-11-24 16:28:41] - dev/eu-west-1/infra/kms-environment-key - No updates to perform.
[2022-11-24 16:28:41] - dev/eu-west-1/infra/route53-private-hosted-zone dccitemplate-dev-eu-west-1-infra-route53-private-hosted-zone AWS::CloudFormation::Stack UPDATE_IN_PROGRESS User Initiated
[2022-11-24 16:28:41] - dev/eu-west-1/infra/kms-environment-key - termination protection set to 'enabled'
[2022-11-24 16:28:45] - dev/eu-west-1/infra/route53-private-hosted-zone PrivateHostedZone AWS::Route53::HostedZone UPDATE_IN_PROGRESS
[2022-11-24 16:29:11] - dev/eu-west-1/infra/route53-private-hosted-zone PrivateHostedZone AWS::Route53::HostedZone UPDATE_COMPLETE
[2022-11-24 16:29:11] - dev/eu-west-1/infra/route53-private-hosted-zone dccitemplate-dev-eu-west-1-infra-route53-private-hosted-zone AWS::CloudFormation::Stack UPDATE_COMPLETE_CLEANUP_IN_PROGRESS
[2022-11-24 16:29:11] - dev/eu-west-1/infra/route53-private-hosted-zone dccitemplate-dev-eu-west-1-infra-route53-private-hosted-zone AWS::CloudFormation::Stack UPDATE_COMPLETE
"An error occurred (AccessDenied) when calling the AssumeRole operation: User: arn:aws:sts::1234567890:assumed-role/AWSReservedSSO_DeveloperAccess_1234xyz/user@xxxx.com is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::1234567890:role/iac-roles/infra/project-dev-iac-infra-deploy-role"

@jfalkenstein
Copy link
Contributor

Thanks for posting those logs. I'm currently trying to figure out the source of this bug. Every time I think I've figured it out, I realize I'm mistaken and that's not a bug. I really wish I could just step through your specific deployment and see what's causing the issue. I don't really have an identical situation where I have an iam role that my environment's user CAN assume but my deployment role cannot assume. That's making it a bit difficult for me to diagnose.

The areas of code I keep coming back around to have to do with:

  • Where the connection manager property on stacks temporarily falls back to having no iam role while resolving the iam_role to avoid an infinite recursion
  • How the ConnectionManager caches sessions and session parameters by stack
  • How the stack_output_external resolver differs from the stack_output resolver in how it accesses values.

The problem is that the call-stack gets rather deep (with some recursion) and there are a few levels of caching that might be causing the problem.

In any case, I'm really struggling to find the problem piece of code. I might have to try to set up an arbitrary, new example similar to yours to see if I can simulate it, if I can. But I'm not sure I could.

Alternatively, if you'd be willing to find a time for us to meet over a google meet or zoom or something, if I could guide you on how to set break points to debug the process, I could "look over your shoulder" and help diagnose this issue by see the actual code path followed. Would you be willing to find some time to do that some time next week?

@Spaideri
Copy link
Author

Hi! I work as a consultant for a +billion enterprise so I need to check if I can share anything from the real environment :(

What I can do right now is to share you the IAM Role I'm using as iam_role above. Maybe you could create a similar role denying your Principal aws:PrincipalArn from assuming it to reproduce the issue?

  IacInfraDeployRole:
    Type: AWS::IAM::Role
    Properties:
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/AdministratorAccess
      MaxSessionDuration: 28800
      RoleName: !Sub ${ProjectName}-${Environment}-iac-infra-deploy-role
      Path: /iac-roles/infra/
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Sid: AllowWhitelist
            Effect: Allow
            Principal:
              AWS: !Sub ${AWS::AccountId}
            Condition:
              ForAnyValue:ArnLike:
                aws:PrincipalArn:
                  - !Sub arn:aws:iam::${AWS::AccountId}:role/aws-reserved/sso.amazonaws.com/eu-west-1/AWSReservedSSO_AWSAdministratorAccess_*
                  - !Sub arn:aws:iam::${AWS::AccountId}:role/aws-reserved/sso.amazonaws.com/eu-west-1/AWSReservedSSO_AWSPowerUserAccess_*
            Action:
              - sts:AssumeRole

@jfalkenstein
Copy link
Contributor

It might take me a week or two to get to this. I'm very interested in figuring out what the issue is here, though. It seems a rather elusive bug that probably has to do with caching at some level.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants