AWS environment variables are not honored #1742

electriquo · 2023-05-23T19:40:17Z

Describe the bug
AWS CLI and AWS SDK honor AWS environment variables, while steampipe does not and displays the error

ERROR: rpc error: code = Unknown desc = operation error Elastic Load Balancing v2: DescribeLoadBalancers, https response error StatusCode: 403, RequestID: 9c80b7d8-6cc7-4d20-ae5d-7955fcde037a, api error AccessDenied: User: arn:aws:sts::1234:assumed-role/foo/i-xxx is not authorized to perform: elasticloadbalancing:DescribeLoadBalancers because no identity-based policy allows the elasticloadbalancing:DescribeLoadBalancers action (SQLSTATE HV000)

Steampipe version (steampipe -v)

$ steampipe --version
Steampipe v0.20.2

Plugin version (steampipe plugin list)

$ steampipe plugin list
+---------------------------------------------+---------+-------------+
| Installed Plugin                            | Version | Connections |
+---------------------------------------------+---------+-------------+
| hub.steampipe.io/plugins/turbot/aws@0.103.0 | 0.103.0 | aws         |
+---------------------------------------------+---------+-------------+

To reproduce

Expose AWS environment variables

$ env | grep -o '^AWS.*='
AWS_DEFAULT_REGION=
AWS_REGION=
AWS_ROLE_ARN=
AWS_WEB_IDENTITY_TOKEN_FILE=
AWS_STS_REGIONAL_ENDPOINTS=

Verify AWS environment variables are working, has ReadOnlyAccess role policy and there are no AWS CLI credential file

$ ls -x1R ~/.aws/
/root/.aws/:
cli

/root/.aws/cli:
cache

$ aws --version
aws-cli/2.11.21 Python/3.11.3 Linux/5.4.204-113.362.amzn2.x86_64 exe/x86_64.debian.11 prompt/off

$ aws iam list-attached-role-policies --role-name $(aws sts get-caller-identity --query Arn --output text | rev | cut -d / -f 2 | rev)
{
    "AttachedPolicies": [
        {
            "PolicyName": "ReadOnlyAccess",
            "PolicyArn": "arn:aws:iam::aws:policy/ReadOnlyAccess"
        }
    ]
}

Install steampipe aws plugin
Start steampipe dashboard
Open one of the AWS dashboards, e.g. /aws_insights.dashboard.ec2_application_load_balancer_detail

Expected behavior
AWS plugin should honor AWS environment variables.

Additional context

AWS environment variables docs for AWS_ROLE_ARN states:

If defined, this environment variable overrides the value for the profile setting role_arn. You can't specify a role session name as a command line parameter.
https://hub.steampipe.io/plugins/turbot/aws#configuring-aws-credentials

If there is no bug, then any guidance on how to make AWS plugin to honor AWS environment variables will be appreciated.

The text was updated successfully, but these errors were encountered:

cbruno10 · 2023-05-23T20:20:22Z

Hi @foolioo , can you please let us know which AWS environment variables specifically aren't working for you? Is it just AWS_ROLE_ARN, or are there others as well?

Are you running from an EC2 instance, or another environment, like a local laptop?

Also, if you run steampipe query using the same AWS Steampipe connections you're using with that load balance detail page, are you able to run queries like select name from aws_iam_role, select instance_id from aws_ec2_instance, and select name from aws_s3_bucket?

electriquo · 2023-05-23T21:05:32Z

@cbruno10

an you please let us know which AWS environment variables specifically aren't working for you? Is it just AWS_ROLE_ARN, or are there others as well?

I cannot know exactly which environment variable isn't honored by steampipe even when trying to set STEAMPIPLE_LOG_LEVEL to trace. Could you specify explicitly how to get the information that you need?

Are you running from an EC2 instance, or another environment, like a local laptop?

Kindly clarify why this should matter when AWS environment are available for steampipe?

Also, if you run steampipe query using the same AWS Steampipe connections you're using with that load balance detail page, are you able to run queries like select name from aws_iam_role, select instance_id from aws_ec2_instance, and select name from aws_s3_bucket?

I use the steampipe dashboard and that is where the error appears and same as before, it is unclear why it matters but here you go :)

psql (15.3, server 14.2)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, compression: off)
Type "help" for help.

steampipe=> select name from aws_iam_role limit 1;
ERROR:  rpc error: code = Unknown desc = operation error IAM: ListRoles, https response error StatusCode: 403, RequestID: cc000269-3591-4c0d-bdd6-d08bf25b0baf, api error AccessDenied: User: arn:aws:sts::1234:assumed-role/foo/i-xxx is not authorized to perform: iam:ListRoles on resource: arn:aws:iam::1234:role/ because no identity-based policy allows the iam:ListRoles action

I can confirm that not authorized to perform: iam:ListRoles is incorrect since ReadOnlyAccess policy (as shown above) was granted and here is an expended policy list of it that includes IAMReadOnlyAccess policy

$ aws iam list-policies --scope AWS --no-only-attached --query 'Policies[].PolicyName' --no-cli-pager | grep IAMReadOnlyAccess
    "IAMReadOnlyAccess",

electriquo · 2023-05-24T13:24:22Z

@cbruno10 Maybe aws/aws-sdk-go-v2#2867 is the corresponding issue and seems like https://github.com/hasheddan/stack-aws/blob/5680d65ec37d585a3fc5c8f8112d3680b38cccfd/pkg/clients/aws.go#L123-L145 is the solution (aws/aws-sdk-go-v2#475 (comment))

cbruno10 · 2023-05-24T19:20:27Z

@foolioo Can you please share the following information:

What's in your ~/.steampipe/config/aws.spc (with sensitive info removed)?
What's in ~/.aws/credentials and ~/.aws/config (with sensitive info removed again)?
What are the exact steampipe commands you're running when using dashboards or queries?

For the environment variables you listed in your original comment, AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE, I haven't used these before, so seeing how you use them with your AWS credentials and config file would be helpful for us to reproduce the error from our end.

And thanks for sharing those issues. It looks like support for the web identity credential provider was added in AWS SDK Go v2 v0.20.0, and we currently v1.44.189. So I'm not sure if we need the workaround code you linked in the plugin, or if there's something shorter we can add if our plugin is not compatible with that authentication method today.

electriquo · 2023-05-24T21:19:36Z

@cbruno10

What's in your ~/.steampipe/config/aws.spc?

connection "aws" {
  plugin = "aws"
  regions = ["*"]
  profile = "default"
}

What's in ~/.aws/credentials and ~/.aws/config

Both files do not exist. Otherwise I wouldn't report that AWS environment variables are not honored

What are the exact steampipe commands you're running when using dashboards or queries?

steampipe service start --foreground --dashboard --dashboard-listen=network

cbruno10 · 2023-05-25T15:40:49Z

Thanks @foolioo for the additional info, we'll look to try and reproduce from our end using the environment variables you listed above and let you know what we find.

electriquo · 2023-05-28T05:04:36Z

@cbruno10 Where you able to reproduce and find the issue?

cbruno10 · 2023-05-30T13:43:16Z

Hi @foolioo , no not yet, we're still investigating. We'll share more info once we have it.

graza-io · 2023-05-31T10:05:34Z

May not resolve issue but could you maybe try setting the env var AWS_SDK_LOAD_CONFIG=true

I did attempt a basic example of testing environment variables for authentication (access/secret key combination) & that worked out fine.

❯ source .tempenv
❯ ls -x1R ~/.aws/
❯ ll -a ~/.aws
total 0
drwxr-xr-x   2 graza  staff    64B 31 May 10:28 .
drwxr-x---+ 42 graza  staff   1.3K 31 May 10:46 ..
❯ aws --version
aws-cli/2.11.23 Python/3.11.3 Darwin/22.5.0 exe/x86_64 prompt/off
❯ env | grep -o '^AWS.*='
AWS_SDK_LOAD_CONFIG=
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_DEFAULT_REGION=
❯ steampipe query "select count(*) from aws_s3_bucket"
+-------+
| count |
+-------+
| 121   |
+-------+

Also would it be possible to test if writing the env vars to relevant config/credentials files works successfully with Steampipe?

e-gineer · 2023-05-31T10:08:56Z

@foolioo I think you should remove this from your spc file:

profile = "default"

since you don’t have any aws config files defined then I assume you do not have a default profile to reference?

graza-io · 2023-05-31T10:37:58Z

Good catch, I didn't set profile in my aws.spc

❯ cat ~/.steampipe/config/aws.spc
connection "aws" {
  plugin = "aws"
  regions = ["eu-*"]
}

When adding the line back in I do indeed get a (different) but understandable error.

❯ vi ~/.steampipe/config/aws.spc
❯ cat ~/.steampipe/config/aws.spc
connection "aws" {
  plugin = "aws"
  regions = ["eu-*"]
  profile = "default"
}

❯ steampipe query "select count(*) from aws_s3_bucket"

Error: operation error S3: ListBuckets, exceeded maximum number of attempts, 9, failed to sign request: failed to retrieve credentials: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, exceeded maximum number of attempts, 3, request send failed, Get "http://169.254.169.254/latest/meta-data/iam/security-credentials/": dial tcp 169.254.169.254:80: connect: no route to host (SQLSTATE HV000)

+-------+
| count |
+-------+
+-------+

graza-io · 2023-06-12T08:12:00Z

Hey @foolioo - can you check/confirm that removing the profile from your .spc file allowed you to successfully use the environment variables as expected?

electriquo · 2023-06-12T09:06:10Z

@graza-io Confirmed. Removing the profile seems to resolve the issue. Will conduct a deeper testing later on and will reopen the issue if it persists

electriquo added the bug Something isn't working label May 23, 2023

graza-io self-assigned this May 31, 2023

electriquo closed this as completed Jun 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AWS environment variables are not honored #1742

AWS environment variables are not honored #1742

electriquo commented May 23, 2023 •

edited

Loading

cbruno10 commented May 23, 2023

electriquo commented May 23, 2023 •

edited

Loading

electriquo commented May 24, 2023 •

edited

Loading

cbruno10 commented May 24, 2023

electriquo commented May 24, 2023

cbruno10 commented May 25, 2023

electriquo commented May 28, 2023

cbruno10 commented May 30, 2023

graza-io commented May 31, 2023

e-gineer commented May 31, 2023

graza-io commented May 31, 2023

graza-io commented Jun 12, 2023

electriquo commented Jun 12, 2023

AWS environment variables are not honored #1742

AWS environment variables are not honored #1742

Comments

electriquo commented May 23, 2023 • edited Loading

cbruno10 commented May 23, 2023

electriquo commented May 23, 2023 • edited Loading

electriquo commented May 24, 2023 • edited Loading

cbruno10 commented May 24, 2023

electriquo commented May 24, 2023

cbruno10 commented May 25, 2023

electriquo commented May 28, 2023

cbruno10 commented May 30, 2023

graza-io commented May 31, 2023

e-gineer commented May 31, 2023

graza-io commented May 31, 2023

graza-io commented Jun 12, 2023

electriquo commented Jun 12, 2023

electriquo commented May 23, 2023 •

edited

Loading

electriquo commented May 23, 2023 •

edited

Loading

electriquo commented May 24, 2023 •

edited

Loading