Add support for cross account roles

[atward]

This adds the ability to query buckets outside the current account. This is needed as aws-go-sdk has no credentials config support for assuming roles when using instance profiles.

[ribbybibby]

Thanks for the PR @atward!

This is needed as aws-go-sdk has no credentials config support for assuming roles when using instance profiles.

Have you tried configuring a profile as detailed on this page https://docs.aws.amazon.com/cli/latest/topic/config-vars.html?

# In ~/.aws/config
[profile crossaccount]
role_arn=arn:aws:iam:...
credential_source=Ec2InstanceMetadata

I'm mostly just interested as I have ran into the same problem with instance profiles and assumed roles myself in the past and I'd never seen this option before. It looks to me like aws-sdk-go should support it as of this PR: aws/aws-sdk-go#2201.

I don't have quick access to an AWS environment where I can test this, so I can't check it myself right now.

Even if it does work, I do understand that setting up an ~/.aws/config file is probably less desirable than a command line flag in a lot of cases. However, as a rule, I try to avoid adding AWS credential options that can be set outside of the program, so I'm a little hesitant here.

[atward]

I was not aware of pr/2201 I was going off issues/1019 which was still open.

Config works as per the linked python documentation given AWS_SDK_LOAD_CONFIG=true AWS_PROFILE=crossaccount

[atward]

I spoke too soon.

When I compile s3_exporter it works as described. Using the pre-compiled 0.1.0 binary results in AccessDenied.

I'm unsure what's causing the problem but could we get an updated release?

[atward]

Logs

go get github.com/ribbybibby/s3_exporter
cd ~/go/src/github.com/ribbybibby/s3_exporter/
make

>> checking code style
mkdir -p /root/go/bin
curl -s -L https://github.com/dominikh/go-tools/releases/download/2019.1/staticcheck_linux_amd64 > /root/go/bin/staticcheck
>> running staticcheck
chmod +x /root/go/bin/staticcheck
GO111MODULE=on go list -e -compiled -test=true -export=false -deps=true -find=false -tags= -- ./... > /dev/null
go: finding github.com/prometheus/common v0.3.0
go: finding github.com/prometheus/client_golang v0.9.2
go: finding github.com/aws/aws-sdk-go v1.19.11
go: finding golang.org/x/text v0.3.0
go: finding gopkg.in/alecthomas/kingpin.v2 v2.2.6
go: finding github.com/prometheus/client_golang v0.9.1
go: finding github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223
go: finding github.com/go-kit/kit v0.8.0
go: finding github.com/matttproud/golang_protobuf_extensions v1.0.1
go: finding golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5
go: finding gopkg.in/yaml.v2 v2.2.1
go: finding github.com/gogo/protobuf v1.1.1
go: finding golang.org/x/sync v0.0.0-20181108010431-42b317875d0f
go: finding github.com/golang/protobuf v1.2.0
go: finding github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973
go: finding github.com/julienschmidt/httprouter v1.2.0
go: finding github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910
go: finding github.com/prometheus/procfs v0.0.0-20181005140218-185b4288413d
go: finding github.com/prometheus/procfs v0.0.0-20181204211112-1dc9a6cbc91a
go: finding github.com/pkg/errors v0.8.0
go: finding github.com/prometheus/common v0.0.0-20181126121408-4724e9255275
go: finding golang.org/x/net v0.0.0-20181201002055-351d144fa1fc
go: finding github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc
go: finding github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515
go: finding golang.org/x/net v0.0.0-20181114220301-adae6a3d119a
go: finding github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf
go: finding github.com/sirupsen/logrus v1.2.0
go: finding github.com/go-logfmt/logfmt v0.3.0
go: finding github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af
go: finding github.com/go-stack/stack v1.8.0
go: finding gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405
go: finding golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33
go: finding github.com/davecgh/go-spew v1.1.1
go: finding github.com/konsorten/go-windows-terminal-sequences v1.0.1
go: finding github.com/pmezard/go-difflib v1.0.0
go: finding github.com/stretchr/testify v1.2.2
go: finding github.com/stretchr/objx v0.1.1
go: finding golang.org/x/crypto v0.0.0-20180904163835-0709b304e793
go: downloading github.com/aws/aws-sdk-go v1.19.11
go: downloading github.com/prometheus/common v0.3.0
go: downloading github.com/prometheus/client_golang v0.9.2
go: downloading gopkg.in/alecthomas/kingpin.v2 v2.2.6
go: downloading github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc
go: downloading github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf
go: downloading github.com/sirupsen/logrus v1.2.0
go: downloading golang.org/x/crypto v0.0.0-20180904163835-0709b304e793
go: downloading github.com/golang/protobuf v1.2.0
go: downloading github.com/prometheus/procfs v0.0.0-20181204211112-1dc9a6cbc91a
go: downloading github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973
go: downloading github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910
go: downloading github.com/matttproud/golang_protobuf_extensions v1.0.1
go: downloading golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5
go: downloading github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af
GO111MODULE=on /root/go/bin/staticcheck -ignore "" ./...
>> running check for unused/missing packages in go.mod
GO111MODULE=on go mod tidy
go: downloading github.com/stretchr/testify v1.2.2
go: downloading golang.org/x/net v0.0.0-20181201002055-351d144fa1fc
go: downloading golang.org/x/sync v0.0.0-20181108010431-42b317875d0f
go: downloading github.com/konsorten/go-windows-terminal-sequences v1.0.1
go: downloading github.com/pmezard/go-difflib v1.0.0
go: downloading github.com/davecgh/go-spew v1.1.1
go: downloading golang.org/x/text v0.3.0
>> running check for unused packages in vendor/
GO111MODULE=on go mod vendor
curl -s -L https://github.com/prometheus/promu/releases/download/v0.2.0/promu-0.2.0.linux-amd64.tar.gz | tar -xvzf - -C /tmp/tmp.hAlGG1rwP9
promu-0.2.0.linux-amd64/
promu-0.2.0.linux-amd64/LICENSE
promu-0.2.0.linux-amd64/NOTICE
promu-0.2.0.linux-amd64/promu
mkdir -p /root/go/bin
cp /tmp/tmp.hAlGG1rwP9/promu-0.2.0.linux-amd64/promu /root/go/bin/promu
rm -r /tmp/tmp.hAlGG1rwP9
>> building binaries
GO111MODULE=on /root/go/bin/promu build --prefix /root/go/src/github.com/ribbybibby/s3_exporter
 >   s3_exporter

$ AWS_REGION=ap-southeast-2 AWS_SDK_LOAD_CONFIG=true AWS_PROFILE=crossaccount ./s3_exporter --log.level=debug 
INFO[0000] Starting s3_exporter (version=, branch=, revision=)  source="s3_exporter.go:164"
INFO[0000] Build context (go=go1.11.5, user=, date=)     source="s3_exporter.go:165"
INFO[0000] Listening on :9340                            source="s3_exporter.go:182"

When using precompiled binaries v0.1.0

$ AWS_REGION=ap-southeast-2 AWS_SDK_LOAD_CONFIG=true AWS_PROFILE=crossaccount /usr/local/sbin/s3_exporter --log.level=debug
INFO[0000] Starting s3_exporter (version=, branch=, revision=)  source="s3_exporter.go:164"
INFO[0000] Build context (go=go1.8.3, user=, date=)      source="s3_exporter.go:165"
INFO[0000] Listening on :9340                            source="s3_exporter.go:182"
ERRO[0012] AccessDenied: Access Denied
	status code: 403, request id: 32DBLAHA0F, host id: WA2nTjs/STUFFKD5Ph8qk=  source="s3_exporter.go:82"

make tarball attached for comparison: s3_exporter-0.1.0.linux-amd64.tar.gz

[ribbybibby]

Hi @atward. I've released 0.2.0 - can you please check if it works for you.

[atward]

Closed by v0.2.0 release

  Service s3_exporter-crossaccount
     ✔  should be running
  File /etc/systemd/system/s3_exporter-crossaccount.service
     ✔  should exist
     ✔  content should include "AWS_REGION=ap-southeast-2"
     ✔  content should include "AWS_SDK_LOAD_CONFIG=true"
     ✔  content should include "AWS_PROFILE=crossaccount"
  Processes s3_exporter
     ✔  entries.length should eq 1
     ✔  commands.first should include "--web.listen-address localhost:9342"
  http GET on localhost:9342/metrics
     ✔  status should eq 200
     ✔  body should include "go_memstats"
  http GET on localhost:9342/probe
     ✔  status should eq 200
     ✔  body should include "s3_list_success{bucket=\"\",prefix=\"\"} 0"
  http GET on localhost:9342/probe?bucket=mybucket
     ✔  status should eq 200
     ✔  body should include "s3_list_success{bucket=\"mybucket\",prefix=\"\"} 1"