Unable to find S3 information

[thoellrich]

Using rev bacfbc163e1574055d38bc3d5f4e13cd3163c484, awless does not seem to find any S3 objects in my account. awscli and awless use the same access_key/secret_key/region:

$ aws s3 ls | wc -l
138
$ aws s3 ls | grep debug
2016-07-18 22:47:00 debugging-log-elb

$ awless sync -e -v
[info] running sync: fetching remote resources for local store
[extra] sync: fetched access service took 915.436447ms
[extra] sync: fetched storage service took 1.173232861s
[extra] sync: fetched infra service took 1.586694499s
[info] -> access: 28 roles, 12 groups, 34 policies, 63 users
[info] -> storage: 0 bucket, 0 storageobject
[info] -> infra: 67 instances, 20 keypairs, 112 securitygroups, 123 volumes, 26 subnets, 6 vpcs, 6 internetgateways, 18 routetables
[info] sync took 2.037318521s
$ awless show -e -v @debugging-log-elb
[info] cannot resolve resource - running full sync
[extra] sync: fetched access service took 860.4982ms
[extra] sync: fetched infra service took 1.107564767s
[extra] sync: fetched storage service took 1.243121296s
[info] resource with id @debugging-log-elb not found

[fxaguessy]

Is your current region, the one in which the buckets are ? Since 5bfa794 we only fetch the buckets (and the related objects) that are in the current region. This is not the behaviour of aws CLI.

[thoellrich]

Yes, all buckets are in the same region as the current region (us-east-1).

$ awless config list | grep region
region: us-east-1       (string)
$ grep region ~/.aws/config
region = us-east-1

In particular the sample bucket above in the AWS console:

[fxaguessy]

That's strange.
We are more close to the aws command aws s3api list-buckets . Could you try this command to see if you have the same results as with aws s3 ls ?

[thoellrich]

Yes, almost same result - the OWNER is also listed using list-buckets which adds 1 line compared to aws s3 ls:

$ aws s3api list-buckets --output text | wc -l
139
$ aws s3api list-buckets --output text | grep debug
BUCKETS 2016-07-18T22:47:00.000Z        debugging-log-elb
$ 

[thoellrich]

I guess this is also of interest:

$ aws s3api get-bucket-location --bucket debugging-log-elb
{
    "LocationConstraint": null
}
$

Maybe 5bfa794#diff-e756db2d4011e6ff8d99cff1d90ad3aaR233 needs to include "" or nil?

Edit: guess it's a bit more complicated. According to http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGETlocation.html you need to consider the following value: empty string (for the US East (N. Virginia) region)

[fxaguessy]

Thanks for pointing the bug. It should be that. Could you test if ecfddff fix that ?

[thoellrich]

Bingo! This seems to have fixed it. Thank you.

$ awless sync -e -v
[info] running sync: fetching remote resources for local store
[extra] sync: fetched access service took 911.829818ms
[extra] sync: fetched infra service took 1.617184313s
[extra] sync: fetched storage service took 21.001531525s
[info] -> infra: 18 routetables, 6 vpcs, 123 volumes, 6 internetgateways, 67 instances, 26 subnets, 20 keypairs, 112 securitygroups
[info] -> storage: 137 buckets, 33646 storageobjects
[info] -> access: 34 policies, 63 users, 38 roles, 12 groups
[info] sync took 59.813545313s
$

[fxaguessy]

Great, thanks for finding the source of the bug.
The sync seems quite slow however, don't hesitate to use listing commands with --local until we find ways to make things faster.