ICE access data in S3 #29
Comments
|
with the above config, i get error: |
|
Hi @kamal2222ahmed. You should give the bucket name instead of the ARN: |
|
here is my ice/assets/ice.properties whether or not to start processorice.processor=true whether or not to start reader/UIice.reader=true whether or not to start reservation capacity pollerice.reservationCapacityPoller=false default reservation period, possible values are oneyear, threeyearice.reservationPeriod=threeyear default reservation utilization, possible values are LIGHT, MEDIUM, HEAVY. If you have both (LIGHT or MEDIUM) and HEAVY RIs, make sure youdo not put HEAVY here. the highstock url; host it somewhere else and change this if you need HTTPSice.highstockUrl=https://code.highcharts.com/stock/4.2.1/highstock.js url prefix, e.g. http://ice.netflix.com/. Will be used in alert emails.ice.urlPrefix= from email addressice.fromEmail= ec2 ondemand hourly cost threshold to send alert email. The alert email will be sent at most once per day.ice.ondemandCostAlertThreshold=250 ec2 ondemand hourly cost alert emails, separated by ","ice.ondemandCostAlertEmails= modify the following 5 properties according to your billing files configuration. if you have multiple payer accounts, you will need to specify multiple values for each property. s3 bucket name where the billing files are. multiple bucket names are delimited by ",". Ice must have read access to billing s3 bucket.ice.billing_s3bucketname=repl-xxxxxxxxxx-yyyy location for the billing bucket. It should be specified for buckets using v4 validationice.billing_s3bucketregion=us-east-1 prefix of the billing files. multiple prefixes are delimited by ","ice.billing_s3bucketprefix=repl/ specify your payer account id here if across-accounts IAM role access is used. multiple account ids are delimited by ",". "ice.billing_payerAccountId=,222222222222" means assumed role access is only used for the second bucket. specify the assumed role name here if you use IAM role access to read from billing s3 bucket. multiple role names are delimited by ",". "ice.billing_accessRoleName=,ice" means assumed role access is only used for the second bucket. specify external id here if it is used. multiple external ids are delimited by ",". if you don't use external id, you can leave this property unset. specify your custom tags here. Multiple tags are delimited by ",". If specified, BasicResourceService will be used to generate resource groups for you. PLEASE MAKE SURE you have limited number (e.g. < 100) of unique value combinations from your custom tags, otherwise Ice performance will begreatly affected. start date in millis from when you want to start processing the billing filesice.startmillis=1364774400000 you company name. it will be used by UIice.companyName=AAAA s3 bucket name where Ice can store output files. Ice must have read and write access to billing s3 bucket.ice.work_s3bucketname=repl-xxxxxxxxxxxxxxx-yyyy prefix of Ice output filesice.work_s3bucketprefix=repl/ local directory for Ice processor. the directory must exist.ice.processor.localDir=/mnt/ice_processor local directory for Ice reader. the directory must exist.ice.reader.localDir=/mnt/ice_reader monthly data cache size for Ice reader.ice.monthlycachesize=12 change the follow account settingsice.account.account1=payer account number set reservation owner accounts. "ice.owneraccount.account2=account3,account4" means reservations in account2 can be shared by account3 andaccount4 if reservation capacity poller is enabled, the poller will try to poll reservation capacity through ec2 API (desribeReservedInstances) foreach reservation owner account. if reservation capacity poller needs to use IAM role to access ec2 API, set the assumed role here for each reservation owner accountice.owneraccount.account1.role=ice if reservation capacity poller needs to use IAM role to access ec2 API and external id is used, set the external id here for each reservation owner account. otherwise you can leave it unset. and when i run: docker-compose up i get error: | 2017-10-17 15:04:40,932 [localhost-startStop-1] INFO core.StandardContext - Unable to set the web application class loader property [clearReferencesStatic] to [false] as the property does not exist. |
|
also, when i try to access my account at https://console.aws.amazon.com/billing/home?#/account Now, would a payer account id work ? or do we need to have individual account IDs for ICE to work ? |
|
Hi @kamal2222ahmed <https://github.com/kamal2222ahmed>,
The error message says that Ice fails to download reservation price.
Given the s3 bucket ARN, can ICE extract the data?
Yes. But Ice needs to have the IAM credentials (AK/SK) for an IAM user that
has the rights specified in the documentation
<https://github.com/Teevity/ice>. Check section "Example IAM Permissions"
on this page and ensure that your IAM user has the appropriate rights.
Nicolas
…--
Nicolas Fonrose | Teevity | Founder
+33.6.61.35.43.31
https://teevity.com - Cloud Costs Analytics built on NetflixOSS
twitter - @nfonrose / @Teevity
On Tue, Oct 17, 2017 at 9:22 PM, kamal2222ahmed ***@***.***> wrote:
also, when i try to access my account at https://console.aws.amazon.
com/billing/home?#/account
i get error:
You Need Permissions
You don't have permission to access billing information for this account.
Contact your AWS administrator if you need help. If you are an AWS
administrator, you can provide permissions for your users or groups by
making sure that (1) this account allows IAM and federated users to access
billing information and (2) you have the required IAM permissions.
Now, would a payer account id work ? or do we need to have individual
account IDs for ICE to work ?
basically:
Given the s3 bucket ARN, can ICE extract the data?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#29 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AACUpdbSHfeJFqzJ8DZ7wjpD20DAXumDks5stP6HgaJpZM4P8bFv>
.
|
|
Hi Nicolas, | 2017-10-20 13:12:39,175 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - trying to download cloudcheckr/NNNNNNNNNN-aws-billing-detailed-line-items-with-resources-and-tags-2016-10.csv.zip... Any idea how i can resolve UnknownHostException ? |
|
i have tried on another host setup for docker, and i got: Server running. Browse to http://localhost:8080/ice Caused by UnknownHostException: local-ice-xxxxxxx-work-bucket.s3-us-east-1.amazonaws.com |
|
Hi,
Are you sure local-ice-xxxxxxx-work-bucket.s3-us-east-1.amazonaws.com matches a real bucket on your account ?
Is it the exact name ? Good region ?
Do you manage to list the content of this bucket using the AWS CLI with the same credentials Ice is using ?
Nicolas
…Sent from my phone
Le 21 oct. 2017 à 16:40, kamal2222ahmed ***@***.***> a écrit :
i have tried on another host setup for docker, and i got:
Server running. Browse to http://localhost:8080/ice
ice_1 | | Error 2017-10-21 10:07:10,610 [com.netflix.ice.processor.BillingFileProcessor] ERROR processor.BillingFileProcessor - Error reading from file lastProcessMillis_2016-10
ice_1 | Message: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: 61D7DF788A11FD3C)
ice_1 | Line | Method
ice_1 | ->> 1588 | handleErrorResponse in com.amazonaws.http.AmazonHttpClient$RequestExecutor
ice_1 | - - -
Caused by UnknownHostException: local-ice-xxxxxxx-work-bucket.s3-us-east-1.amazonaws.com
ice_1 | ->> 1280 | getAllByName0 in java.net.InetAddress
ice_1 | -
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
Hi Nicolas, $ aws s3 ls $aws s3 ls repl-xxxxxxxxxxxx-backup stdout: and these are the same credentials i used as the one i use with ice. now when i run: ./grailsw -Djava.net.preferIPv4Stack=true -Dice.s3AccessKeyId= ACCESSKEY -Dice.s3SecretKey=SECRETKEY run-app Message: Unable to execute HTTP request: repl-XXXXXXXXXXX-YYYYY.s3-us-east-1.amazonaws.com | 1035 | executeHelper in '' Caused by UnknownHostException: repl-XXXXXXXXXX-YYYYYY.s3-us-east-1.amazonaws.com | 1185 | getAllByName in '' |
|
FWIW, I fixed the issue by setting ice.billing_s3bucketregion= Yes, thats right blank. The problem seems to be when you define a region, it uses the URL of epl-XXXXXXXXXXX-YYYYY.s3-us-east-1.amazonaws.com, it should be epl-XXXXXXXXXXX-YYYYY.s3.us-east-1.amazonaws.com Notice that it's doing s3-us-east-1 instead of s3.us-east-1. Setting it to a blank var makes it work fine in us-east-1 since s3.amazonaws.com works fine in that region |
|
@kamal2222ahmed Can you confirm if there is a bug that's adding a dash instead of a period? |
|
After figuring that out on my own, the issue is confirmed: |
|
Closing, bug in ice. |
|
We could add a condition so that, when people specify a region when the
bucket is us-east-1, it gets ignored when building the S3 endpoint.
But as @Juberstine was pointing out, user can also just leave the region
property empty in this case.
I guess a comment in the example configuration file could make this clearer.
…--
Nicolas Fonrose | Teevity | Founder
+33.6.61.35.43.31
https://teevity.com - Cloud Costs Analytics built on NetflixOSS
twitter - @nfonrose / @Teevity
On Thu, Sep 20, 2018 at 6:13 PM Jonathon Brouse ***@***.***> wrote:
Closing, bug in ice.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#29 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AACUpRLzVYZZOqkjcYXF3nclsxSErQZEks5uc76RgaJpZM4P8bFv>
.
|
I have -
Access Key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Secret Key: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
bucketname: arn:aws:s3:::unique_bucketname
All cost data goes to a combined billing report in S3. I have access to the s3 bucket containing the files.
based on the Access and Secret key.
Given the s3 bucket ARN, can ICE extract the data? Or it needs the account ID, or something else.
The text was updated successfully, but these errors were encountered: