Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error while running the report #37

Open
sathyajackson opened this issue Jan 7, 2019 · 20 comments
Open

Error while running the report #37

sathyajackson opened this issue Jan 7, 2019 · 20 comments

Comments

@sathyajackson
Copy link

Hello,

Sorry for very late responds, I have hooked up some other project. Alright, here we go, I used the classic version of the run.py and I end up with below error

[ error ]
Traceback (most recent call last):
File "src/make_xlsx.py", line 630, in
main(name)
File "src/make_xlsx.py", line 612, in main
gen_weekly_variations(workbook, header_format, val_format)
File "src/make_xlsx.py", line 201, in gen_weekly_variations
with open(IN_ABSOLUTE_COST_PER_MONTH) as f:
FileNotFoundError: [Errno 2] No such file or directory: 'out/absolute.csv'
Any suggestions would be of great help.

@sathyajackson
Copy link
Author

any update would be of great help

@thibautcornolti
Copy link
Member

Hello,

Can I have more logs and the exact command you typed?

Thanks.

@sathyajackson
Copy link
Author

sathyajackson commented Jan 8, 2019

here you go.

Command:

./run.py --billing default billing-aws-cost-report billing/XXX.csv --ec2 default --xlsx-name billing.

Here is the complete log.

 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  582M  100  582M    0     0  41.3M      0  0:00:14  0:00:14 --:--:-- 43.2M
Download billings for default...
Fetching ec2 data for all accounts in ap-south-1...
Fetching ec2 data for all accounts in eu-west-3...
Fetching ec2 data for all accounts in eu-north-1...
Fetching ec2 data for all accounts in eu-west-2...
Fetching ec2 data for all accounts in eu-west-1...
Fetching ec2 data for all accounts in ap-northeast-2...
Fetching ec2 data for all accounts in ap-northeast-1...
Fetching ec2 data for all accounts in sa-east-1...
Fetching ec2 data for all accounts in ca-central-1...
Fetching ec2 data for all accounts in ap-southeast-1...
Fetching ec2 data for all accounts in ap-southeast-2...
Fetching ec2 data for all accounts in eu-central-1...
Fetching ec2 data for all accounts in us-east-1...
Fetching ec2 data for all accounts in us-east-2...
Fetching ec2 data for all accounts in us-west-1...
Fetching ec2 data for all accounts in us-west-2...
[default - ap-northeast-1] Getting reserved instances...
[default - ap-northeast-2] Getting reserved instances...
[default - ap-south-1] Getting reserved instances...
[default - eu-north-1] Getting reserved instances...
[default - eu-west-1] Getting reserved instances...
[default - ap-southeast-1] Getting reserved instances...
[default - eu-west-2] Getting reserved instances...
[default - eu-west-3] Getting reserved instances...
[default - ca-central-1] Getting reserved instances...
[default - us-east-1] Getting reserved instances...
[default - sa-east-1] Getting reserved instances...
[default - us-west-1] Getting reserved instances...
[default - us-east-2] Getting reserved instances...
[default - us-west-2] Getting reserved instances...
[default - eu-central-1] Getting reserved instances...
[default - ap-southeast-2] Getting reserved instances...
[default - us-east-1] Getting on-demand instances...
[default - ca-central-1] Getting on-demand instances...
[default - us-east-2] Getting on-demand instances...
[global - ca-central-1] Getting offerings for all instances...
[global - us-east-2] Getting offerings for all instances...
[default - eu-west-1] Getting on-demand instances...
[global - ca-central-1] Matching on-demand instances with reserved instances...
[global - ca-central-1] Done!
[default - eu-west-2] Getting on-demand instances...
[global - us-east-2] Matching on-demand instances with reserved instances...
[global - us-east-2] Done!
[default - eu-west-3] Getting on-demand instances...
[global - eu-west-1] Getting offerings for all instances...
[default - us-west-1] Getting on-demand instances...
[default - eu-north-1] Getting on-demand instances...
[global - eu-west-2] Getting offerings for all instances...
[default - us-west-2] Getting on-demand instances...
[default - eu-central-1] Getting on-demand instances...
[global - eu-west-1] Matching on-demand instances with reserved instances...
[global - eu-west-1] Done!
[global - eu-west-2] Matching on-demand instances with reserved instances...
[global - eu-west-2] Done!
[default - ap-northeast-1] Getting on-demand instances...
[global - eu-west-3] Getting offerings for all instances...
[global - us-west-1] Getting offerings for all instances...
[global - us-west-2] Getting offerings for all instances...
[global - eu-north-1] Getting offerings for all instances...
[global - eu-west-3] Matching on-demand instances with reserved instances...
[global - eu-west-3] Done!
[global - eu-central-1] Getting offerings for all instances...
[global - us-east-1] Getting offerings for all instances...
[default - us-east-1] Getting offerings for instance 1/8...
[global - us-west-1] Matching on-demand instances with reserved instances...
[global - us-west-1] Done!
[default - sa-east-1] Getting on-demand instances...
[default - ap-south-1] Getting on-demand instances...
[global - us-west-2] Matching on-demand instances with reserved instances...
[global - us-west-2] Done!
[global - eu-north-1] Matching on-demand instances with reserved instances...
[global - eu-north-1] Done!
[global - eu-central-1] Matching on-demand instances with reserved instances...
[global - eu-central-1] Done!
[default - ap-northeast-2] Getting on-demand instances...
[global - ap-northeast-1] Getting offerings for all instances...
[global - ap-northeast-1] Matching on-demand instances with reserved instances...
[global - ap-northeast-1] Done!
[default - ap-southeast-1] Getting on-demand instances...
[default - ap-southeast-2] Getting on-demand instances...
[global - sa-east-1] Getting offerings for all instances...
[global - ap-south-1] Getting offerings for all instances...
[global - sa-east-1] Matching on-demand instances with reserved instances...
[global - sa-east-1] Done!
[global - ap-northeast-2] Getting offerings for all instances...
[global - ap-south-1] Matching on-demand instances with reserved instances...
[global - ap-south-1] Done!
[global - ap-northeast-2] Matching on-demand instances with reserved instances...
[global - ap-northeast-2] Done!
[global - ap-southeast-2] Getting offerings for all instances...
[global - ap-southeast-1] Getting offerings for all instances...
[global - ap-southeast-2] Matching on-demand instances with reserved instances...
[global - ap-southeast-2] Done!
[global - ap-southeast-1] Matching on-demand instances with reserved instances...
[global - ap-southeast-1] Done!
Fetched ec2 data for all accounts in ap-south-1
Fetched ec2 data for all accounts in eu-west-3
Fetched ec2 data for all accounts in eu-north-1
Fetched ec2 data for all accounts in eu-west-2
Fetched ec2 data for all accounts in eu-west-1
Fetched ec2 data for all accounts in ap-northeast-2
Fetched ec2 data for all accounts in ap-northeast-1
Fetched ec2 data for all accounts in sa-east-1
Fetched ec2 data for all accounts in ca-central-1
Fetched ec2 data for all accounts in ap-southeast-1
Fetched ec2 data for all accounts in ap-southeast-2
Fetched ec2 data for all accounts in eu-central-1
[default - us-east-1] Getting offerings for instance 2/8...
[default - us-east-1] Getting offerings for instance 3/8...
[default - us-east-1] Getting offerings for instance 4/8...
[default - us-east-1] Getting offerings for instance 5/8...
[default - us-east-1] Getting offerings for instance 6/8...
[default - us-east-1] Getting offerings for instance 7/8...
[default - us-east-1] Getting offerings for instance 8/8...
[global - us-east-1] Matching on-demand instances with reserved instances...
[global - us-east-1] Done!
Fetched ec2 data for all accounts in us-east-1
Fetched ec2 data for all accounts in us-east-2
Fetched ec2 data for all accounts in us-west-1
Fetched ec2 data for all accounts in us-west-2
Fetching ec2 metadata for default in ap-south-1...
Fetching ec2 metadata for default in eu-west-3...
Fetching ec2 metadata for default in eu-north-1...
Fetching ec2 metadata for default in eu-west-2...
Fetching ec2 metadata for default in eu-west-1...
Fetching ec2 metadata for default in ap-northeast-2...
Fetching ec2 metadata for default in ap-northeast-1...
Fetching ec2 metadata for default in sa-east-1...
Fetching ec2 metadata for default in ca-central-1...
Fetching ec2 metadata for default in ap-southeast-1...
Fetching ec2 metadata for default in ap-southeast-2...
Fetching ec2 metadata for default in eu-central-1...
Fetching ec2 metadata for default in us-east-1...
Fetching ec2 metadata for default in us-east-2...
Fetching ec2 metadata for default in us-west-1...
Fetching ec2 metadata for default in us-west-2...
[574925386644 - ap-southeast-1] Getting instances metadata...
[574925386644 - sa-east-1] Getting instances metadata...
[574925386644 - eu-west-2] Getting instances metadata...
[574925386644 - us-east-1] Getting instances metadata...
[574925386644 - eu-west-1] Getting instances metadata...
[574925386644 - ap-southeast-2] Getting instances metadata...
[574925386644 - us-east-2] Getting instances metadata...
[574925386644 - ap-south-1] Getting instances metadata...
[574925386644 - eu-central-1] Getting instances metadata...
[574925386644 - eu-west-3] Getting instances metadata...
[574925386644 - ap-northeast-1] Getting instances metadata...
[574925386644 - us-west-1] Getting instances metadata...
[574925386644 - us-east-2] Done!
[574925386644 - eu-north-1] Getting instances metadata...
[574925386644 - ca-central-1] Getting instances metadata...
[574925386644 - us-west-2] Getting instances metadata...
[574925386644 - us-east-1] Done!
[574925386644 - ca-central-1] Done!
[574925386644 - eu-west-1] Done!
[574925386644 - eu-west-2] Done!
[574925386644 - ap-northeast-2] Getting instances metadata...
[574925386644 - us-west-1] Done!
[574925386644 - eu-central-1] Done!
[574925386644 - eu-west-3] Done!
[574925386644 - us-west-2] Done!
[574925386644 - sa-east-1] Done!
[574925386644 - eu-north-1] Done!
[574925386644 - ap-northeast-1] Done!
[574925386644 - ap-southeast-2] Done!
[574925386644 - ap-southeast-1] Done!
[574925386644 - ap-northeast-2] Done!
[574925386644 - ap-south-1] Done!
Fetched ec2 metadata for default in ap-south-1
Fetched ec2 metadata for default in eu-west-3
Fetched ec2 metadata for default in eu-north-1
Fetched ec2 metadata for default in eu-west-2
Fetched ec2 metadata for default in eu-west-1
Fetched ec2 metadata for default in ap-northeast-2
Fetched ec2 metadata for default in ap-northeast-1
Fetched ec2 metadata for default in sa-east-1
Fetched ec2 metadata for default in ca-central-1
Fetched ec2 metadata for default in ap-southeast-1
Fetched ec2 metadata for default in ap-southeast-2
Fetched ec2 metadata for default in eu-central-1
Fetched ec2 metadata for default in us-east-1
Fetched ec2 metadata for default in us-east-2
Fetched ec2 metadata for default in us-west-1
Fetched ec2 metadata for default in us-west-2
Processing billing data (1/5 - billing diff)...
Traceback (most recent call last):
  File "src/get_bill_diff.py", line 65, in <module>
    first_month = preserved_months[0]
IndexError: list index out of range
Processing billing data (2/5 - instance history)...
Processing billing data (3/5 - ec2 last month)...
Processing billing data (4/5 - ebs last month)...
Processing billing data (5/5 - s3 cost)...
Generating xlsx file...
Traceback (most recent call last):
  File "src/make_xlsx.py", line 630, in <module>
    main(name)
  File "src/make_xlsx.py", line 612, in main
    gen_weekly_variations(workbook, header_format, val_format)
  File "src/make_xlsx.py", line 201, in gen_weekly_variations
    with open(IN_ABSOLUTE_COST_PER_MONTH) as f:
FileNotFoundError: [Errno 2] No such file or directory: 'out/absolute.csv'

@sathyajackson
Copy link
Author

The bill file I am using is more than 30 G and more than 3 million records.

@thibautcornolti
Copy link
Member

thibautcornolti commented Jan 8, 2019

I guess your bucket is billing-aws-cost-report, is it a bucket with a report enabled?
You can check that here https://console.aws.amazon.com/billing/home?#/reports

You should see something as below:
image

@sathyajackson
Copy link
Author

sathyajackson commented Jan 8, 2019 via email

@thibautcornolti
Copy link
Member

thibautcornolti commented Jan 8, 2019

Got it. So you should have a report enabled on your master account. A report is more than a single .csv, you have to download the entire bucket (sync it with the cli for example).
Are you allowed to do that?

To clarify everything, you can get two types of report with AWS :

Old report will be removed by AWS.
Our tool only works with the new type of report. You have to be sure you're using it.

@sathyajackson
Copy link
Author

sathyajackson commented Jan 8, 2019 via email

@thibautcornolti
Copy link
Member

thibautcornolti commented Jan 8, 2019

Sorry you can't, your file is probably in the old format (typically one .csv.zip generated per month), we do not support that report.
However, you can use our tool without billing, just remove the --billing argument. You just won't see any cost.

@sathyajackson
Copy link
Author

sathyajackson commented Jan 8, 2019 via email

@thibautcornolti
Copy link
Member

Ok! I look forward to your answer.

@sathyajackson
Copy link
Author

sathyajackson commented Jan 9, 2019

I used the following command to execute without --billing and end up with the same error message.

 ./run.py   --ec2 default --xlsx-name bill

Logs

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  585M  100  585M    0     0  39.9M      0  0:00:14  0:00:14 --:--:-- 41.3M
Fetching ec2 data for all accounts in ap-south-1...
Fetching ec2 data for all accounts in eu-west-3...
Fetching ec2 data for all accounts in eu-north-1...
Fetching ec2 data for all accounts in eu-west-2...
Fetching ec2 data for all accounts in eu-west-1...
Fetching ec2 data for all accounts in ap-northeast-2...
Fetching ec2 data for all accounts in ap-northeast-1...
Fetching ec2 data for all accounts in sa-east-1...
Fetching ec2 data for all accounts in ca-central-1...
Fetching ec2 data for all accounts in ap-southeast-1...
Fetching ec2 data for all accounts in ap-southeast-2...
Fetching ec2 data for all accounts in eu-central-1...
Fetching ec2 data for all accounts in us-east-1...
Fetching ec2 data for all accounts in us-east-2...
Fetching ec2 data for all accounts in us-west-1...
Fetching ec2 data for all accounts in us-west-2...
[default - eu-north-1] Getting reserved instances...
[default - eu-west-3] Getting reserved instances...
[default - eu-west-1] Getting reserved instances...
[default - ap-south-1] Getting reserved instances...
[default - eu-west-2] Getting reserved instances...
[default - ap-northeast-1] Getting reserved instances...
[default - ap-northeast-2] Getting reserved instances...
[default - eu-central-1] Getting reserved instances...
[default - ap-southeast-1] Getting reserved instances...
[default - ap-southeast-2] Getting reserved instances...
[default - us-west-2] Getting reserved instances...
[default - us-east-2] Getting reserved instances...
[default - sa-east-1] Getting reserved instances...
[default - ca-central-1] Getting reserved instances...
[default - us-east-1] Getting reserved instances...
[default - us-west-1] Getting reserved instances...
[default - us-east-2] Getting on-demand instances...
[default - ca-central-1] Getting on-demand instances...
[default - us-east-1] Getting on-demand instances...
[global - us-east-2] Getting offerings for all instances...
[default - eu-west-3] Getting on-demand instances...
[global - us-east-2] Matching on-demand instances with reserved instances...
[global - us-east-2] Done!
[default - eu-west-2] Getting on-demand instances...
[global - ca-central-1] Getting offerings for all instances...
[global - ca-central-1] Matching on-demand instances with reserved instances...
[global - ca-central-1] Done!
[default - eu-west-1] Getting on-demand instances...
[default - us-west-2] Getting on-demand instances...
[default - eu-central-1] Getting on-demand instances...
[default - eu-north-1] Getting on-demand instances...
[default - us-west-1] Getting on-demand instances...
[global - eu-west-3] Getting offerings for all instances...
[global - eu-west-1] Getting offerings for all instances...
[default - ap-northeast-1] Getting on-demand instances...
[global - eu-west-3] Matching on-demand instances with reserved instances...
[global - eu-west-3] Done!
[global - eu-west-2] Getting offerings for all instances...
[global - eu-west-1] Matching on-demand instances with reserved instances...
[global - eu-west-1] Done!
[global - us-west-2] Getting offerings for all instances...
[default - ap-south-1] Getting on-demand instances...
[global - eu-north-1] Getting offerings for all instances...
[default - sa-east-1] Getting on-demand instances...
[global - eu-central-1] Getting offerings for all instances...
[global - eu-west-2] Matching on-demand instances with reserved instances...
[global - eu-west-2] Done!
[global - us-west-2] Matching on-demand instances with reserved instances...
[global - us-west-2] Done!
[default - ap-northeast-2] Getting on-demand instances...
[default - ap-southeast-2] Getting on-demand instances...
[global - eu-central-1] Matching on-demand instances with reserved instances...
[global - eu-central-1] Done!
[global - us-east-1] Getting offerings for all instances...
[default - us-east-1] Getting offerings for instance 1/8...
[global - us-west-1] Getting offerings for all instances...
[global - eu-north-1] Matching on-demand instances with reserved instances...
[global - eu-north-1] Done!
[global - sa-east-1] Getting offerings for all instances...
[global - us-west-1] Matching on-demand instances with reserved instances...
[global - us-west-1] Done!
[default - ap-southeast-1] Getting on-demand instances...
[global - ap-northeast-1] Getting offerings for all instances...
[global - ap-northeast-1] Matching on-demand instances with reserved instances...
[global - ap-northeast-1] Done!
[global - sa-east-1] Matching on-demand instances with reserved instances...
[global - sa-east-1] Done!
[global - ap-south-1] Getting offerings for all instances...
[global - ap-south-1] Matching on-demand instances with reserved instances...
[global - ap-south-1] Done!
[global - ap-northeast-2] Getting offerings for all instances...
[global - ap-southeast-2] Getting offerings for all instances...
[global - ap-northeast-2] Matching on-demand instances with reserved instances...
[global - ap-northeast-2] Done!
[global - ap-southeast-2] Matching on-demand instances with reserved instances...
[global - ap-southeast-2] Done!
[global - ap-southeast-1] Getting offerings for all instances...
[global - ap-southeast-1] Matching on-demand instances with reserved instances...
[global - ap-southeast-1] Done!
Fetched ec2 data for all accounts in ap-south-1
Fetched ec2 data for all accounts in eu-west-3
Fetched ec2 data for all accounts in eu-north-1
Fetched ec2 data for all accounts in eu-west-2
Fetched ec2 data for all accounts in eu-west-1
Fetched ec2 data for all accounts in ap-northeast-2
Fetched ec2 data for all accounts in ap-northeast-1
Fetched ec2 data for all accounts in sa-east-1
Fetched ec2 data for all accounts in ca-central-1
Fetched ec2 data for all accounts in ap-southeast-1
Fetched ec2 data for all accounts in ap-southeast-2
Fetched ec2 data for all accounts in eu-central-1
[default - us-east-1] Getting offerings for instance 2/8...
[default - us-east-1] Getting offerings for instance 3/8...
[default - us-east-1] Getting offerings for instance 4/8...
[default - us-east-1] Getting offerings for instance 5/8...
[default - us-east-1] Getting offerings for instance 6/8...
[default - us-east-1] Getting offerings for instance 7/8...
[default - us-east-1] Getting offerings for instance 8/8...
[global - us-east-1] Matching on-demand instances with reserved instances...
[global - us-east-1] Done!
Fetched ec2 data for all accounts in us-east-1
Fetched ec2 data for all accounts in us-east-2
Fetched ec2 data for all accounts in us-west-1
Fetched ec2 data for all accounts in us-west-2
Fetching ec2 metadata for default in ap-south-1...
Fetching ec2 metadata for default in eu-west-3...
Fetching ec2 metadata for default in eu-north-1...
Fetching ec2 metadata for default in eu-west-2...
Fetching ec2 metadata for default in eu-west-1...
Fetching ec2 metadata for default in ap-northeast-2...
Fetching ec2 metadata for default in ap-northeast-1...
Fetching ec2 metadata for default in sa-east-1...
Fetching ec2 metadata for default in ca-central-1...
Fetching ec2 metadata for default in ap-southeast-1...
Fetching ec2 metadata for default in ap-southeast-2...
Fetching ec2 metadata for default in eu-central-1...
Fetching ec2 metadata for default in us-east-1...
Fetching ec2 metadata for default in us-east-2...
Fetching ec2 metadata for default in us-west-1...
Fetching ec2 metadata for default in us-west-2...
[574925386644 - ap-south-1] Getting instances metadata...
[574925386644 - eu-west-1] Getting instances metadata...
[574925386644 - eu-north-1] Getting instances metadata...
[574925386644 - eu-west-3] Getting instances metadata...
[574925386644 - eu-west-2] Getting instances metadata...
[574925386644 - ap-northeast-1] Getting instances metadata...
[574925386644 - ca-central-1] Getting instances metadata...
[574925386644 - sa-east-1] Getting instances metadata...
[574925386644 - ap-southeast-1] Getting instances metadata...
[574925386644 - eu-central-1] Getting instances metadata...
[574925386644 - ap-southeast-2] Getting instances metadata...
[574925386644 - us-west-1] Getting instances metadata...
[574925386644 - us-east-1] Getting instances metadata...
[574925386644 - eu-west-1] Done!
[574925386644 - us-east-2] Getting instances metadata...
[574925386644 - us-west-2] Getting instances metadata...
[574925386644 - ca-central-1] Done!
[574925386644 - us-east-2] Done!
[574925386644 - eu-west-3] Done!
[574925386644 - eu-west-2] Done!
[574925386644 - eu-north-1] Done!
[574925386644 - us-west-1] Done!
[574925386644 - us-east-1] Done!
[574925386644 - eu-central-1] Done!
[574925386644 - ap-south-1] Done!
[574925386644 - us-west-2] Done!
[574925386644 - ap-northeast-2] Getting instances metadata...
Fetched ec2 metadata for default in ap-south-1
Fetched ec2 metadata for default in eu-west-3
Fetched ec2 metadata for default in eu-north-1
Fetched ec2 metadata for default in eu-west-2
Fetched ec2 metadata for default in eu-west-1
[574925386644 - ap-northeast-1] Done!
[574925386644 - sa-east-1] Done!
[574925386644 - ap-southeast-2] Done!
[574925386644 - ap-southeast-1] Done!
[574925386644 - ap-northeast-2] Done!
Fetched ec2 metadata for default in ap-northeast-2
Fetched ec2 metadata for default in ap-northeast-1
Fetched ec2 metadata for default in sa-east-1
Fetched ec2 metadata for default in ca-central-1
Fetched ec2 metadata for default in ap-southeast-1
Fetched ec2 metadata for default in ap-southeast-2
Fetched ec2 metadata for default in eu-central-1
Fetched ec2 metadata for default in us-east-1
Fetched ec2 metadata for default in us-east-2
Fetched ec2 metadata for default in us-west-1
Fetched ec2 metadata for default in us-west-2
Processing billing data (1/5 - billing diff)...
Traceback (most recent call last):
  File "src/get_bill_diff.py", line 65, in <module>
    first_month = preserved_months[0]
IndexError: list index out of range
Processing billing data (2/5 - instance history)...
Processing billing data (3/5 - ec2 last month)...
Processing billing data (4/5 - ebs last month)...
Processing billing data (5/5 - s3 cost)...
Generating xlsx file...
Traceback (most recent call last):
  File "src/make_xlsx.py", line 630, in <module>
    main(name)
  File "src/make_xlsx.py", line 612, in main
    gen_weekly_variations(workbook, header_format, val_format)
  File "src/make_xlsx.py", line 201, in gen_weekly_variations
    with open(IN_ABSOLUTE_COST_PER_MONTH) as f:
FileNotFoundError: [Errno 2] No such file or directory: 'out/absolute.csv'

Correct me if i am worng in using the .run.py command.

@thibautcornolti
Copy link
Member

I just talked with the team:
Finally, we can't run the tool without billing, really sorry for that.

I can help you to setup a report bucket though.

@sathyajackson
Copy link
Author

Hi Thanks for your response, one more quick question, the tool actually use the CPU as the base point for the recommendation, I would like to know if we can have both memory and CPU?

I know that cloud watch will target only of the CPU and not memory ( other than writing custom metrics), but yeh, just thought of check if we can have this included on the script to somehow get the memory information of the machine and based on that providing the recommendation.

@thibautcornolti
Copy link
Member

No you can't use memory to influence recommendations, as you said, it's not supported by CloudWatch and you would need to install a custom software on every single instance.

@sathyajackson
Copy link
Author

alright thanks 👍

@thibautcornolti
Copy link
Member

thibautcornolti commented Jan 9, 2019

Thanks to you! 👍

You can close this issue as soon as you have no more question.

@vmpowercli
Copy link

I get similar error when running the script. Can you please tell me what to enter "--billing profile_name". When I ran the script with --billing default i get the same error message.

Generating xlsx file...
Traceback (most recent call last):
File "src/make_xlsx.py", line 630, in
main(name)
File "src/make_xlsx.py", line 612, in main
gen_weekly_variations(workbook, header_format, val_format)
File "src/make_xlsx.py", line 201, in gen_weekly_variations
with open(IN_ABSOLUTE_COST_PER_MONTH) as f:
FileNotFoundError: [Errno 2] No such file or directory: 'out/absolute.csv'

@thibautcornolti
Copy link
Member

Hello,

$> ./run.py --billing profile_name billing-bucket-name prefix
  • profile_name is default or another profile available in ~/.aws/credentials.
  • billing_bucket_name is you billing bucket. It has to be configured here as following:
    image
  • prefix is prefix put in report config, you can get it by clicking on a report here:
    image
    Prefix here would be bills, billing being the report name.

Thanks.

@AashishChughHome
Copy link

I can see the same error while running the reports, can you please suggest where it's going wrong.
Command I am running-

./run.py --billing EBS esg-ebs-costing ebs-nv --ec2 EBS --xlsx-name EBS

MyError

json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
File "src/get_ec2_data.py", line 73, in
compute_instance_costs = json.load(f)
File "/usr/lib64/python3.6/json/init.py", line 299, in load
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
File "/usr/lib64/python3.6/json/init.py", line 354, in loads
return _default_decoder.decode(s)
File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
File "src/get_ec2_data.py", line 73, in
compute_instance_costs = json.load(f)
File "/usr/lib64/python3.6/json/init.py", line 299, in load
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
File "/usr/lib64/python3.6/json/init.py", line 354, in loads
return _default_decoder.decode(s)
File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
File "src/get_ec2_data.py", line 73, in
compute_instance_costs = json.load(f)
File "/usr/lib64/python3.6/json/init.py", line 299, in load
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
File "/usr/lib64/python3.6/json/init.py", line 354, in loads
return _default_decoder.decode(s)
File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
File "src/get_ec2_data.py", line 73, in
compute_instance_costs = json.load(f)
File "/usr/lib64/python3.6/json/init.py", line 299, in load
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
File "/usr/lib64/python3.6/json/init.py", line 354, in loads
return _default_decoder.decode(s)
File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
File "src/get_ec2_data.py", line 73, in
compute_instance_costs = json.load(f)
File "/usr/lib64/python3.6/json/init.py", line 299, in load
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
File "/usr/lib64/python3.6/json/init.py", line 354, in loads
return _default_decoder.decode(s)
File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
File "src/get_ec2_data.py", line 73, in
compute_instance_costs = json.load(f)
File "/usr/lib64/python3.6/json/init.py", line 299, in load
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
File "/usr/lib64/python3.6/json/init.py", line 354, in loads
return _default_decoder.decode(s)
File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
File "src/get_ec2_data.py", line 73, in
compute_instance_costs = json.load(f)
File "/usr/lib64/python3.6/json/init.py", line 299, in load
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
File "/usr/lib64/python3.6/json/init.py", line 354, in loads
return _default_decoder.decode(s)
File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
File "src/get_ec2_data.py", line 73, in
compute_instance_costs = json.load(f)
File "/usr/lib64/python3.6/json/init.py", line 299, in load
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
File "/usr/lib64/python3.6/json/init.py", line 354, in loads
return _default_decoder.decode(s)
File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
File "src/get_ec2_data.py", line 73, in
compute_instance_costs = json.load(f)
File "/usr/lib64/python3.6/json/init.py", line 299, in load
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
File "/usr/lib64/python3.6/json/init.py", line 354, in loads
return _default_decoder.decode(s)
File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
File "/usr/lib64/python3.6/json/init.py", line 354, in loads
return _default_decoder.decode(s)
File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Fetched ec2 data for all accounts in eu-north-1
Fetched ec2 data for all accounts in ap-south-1
Fetched ec2 data for all accounts in eu-west-3
Fetched ec2 data for all accounts in eu-west-2
Fetched ec2 data for all accounts in eu-west-1
Fetched ec2 data for all accounts in ap-northeast-2
Fetched ec2 data for all accounts in ap-northeast-1
Fetched ec2 data for all accounts in sa-east-1
Fetched ec2 data for all accounts in ca-central-1
Fetched ec2 data for all accounts in ap-southeast-1
Fetched ec2 data for all accounts in ap-southeast-2
Fetched ec2 data for all accounts in eu-central-1
Fetched ec2 data for all accounts in us-east-1
Fetched ec2 data for all accounts in us-east-2
Fetched ec2 data for all accounts in us-west-1
Fetched ec2 data for all accounts in us-west-2
Fetching ec2 metadata for EBS in eu-north-1...
Fetching ec2 metadata for EBS in ap-south-1...
Fetching ec2 metadata for EBS in eu-west-3...
Fetching ec2 metadata for EBS in eu-west-2...
Fetching ec2 metadata for EBS in eu-west-1...
Fetching ec2 metadata for EBS in ap-northeast-2...
Fetching ec2 metadata for EBS in ap-northeast-1...
Fetching ec2 metadata for EBS in sa-east-1...
Fetching ec2 metadata for EBS in ca-central-1...
Fetching ec2 metadata for EBS in ap-southeast-1...
Fetching ec2 metadata for EBS in ap-southeast-2...
Fetching ec2 metadata for EBS in eu-central-1...
Fetching ec2 metadata for EBS in us-east-1...
Fetching ec2 metadata for EBS in us-east-2...
Fetching ec2 metadata for EBS in us-west-1...
Fetching ec2 metadata for EBS in us-west-2...
[954326812311 - eu-north-1] Getting instances metadata...
[954326812311 - ap-south-1] Getting instances metadata...
[954326812311 - eu-west-3] Getting instances metadata...
[954326812311 - eu-north-1] Done!
[954326812311 - eu-west-2] Getting instances metadata...
[954326812311 - eu-west-1] Getting instances metadata...
[954326812311 - eu-west-3] Done!
[954326812311 - ap-south-1] Done!
[954326812311 - ap-northeast-1] Getting instances metadata...
[954326812311 - eu-west-2] Done!
[954326812311 - eu-west-1] Done!
Fetched ec2 metadata for EBS in eu-north-1
[954326812311 - ap-northeast-2] Getting instances metadata...
[954326812311 - sa-east-1] Getting instances metadata...
[954326812311 - ca-central-1] Getting instances metadata...
Fetched ec2 metadata for EBS in ap-south-1
Fetched ec2 metadata for EBS in eu-west-3
[954326812311 - ap-southeast-1] Getting instances metadata...
Fetched ec2 metadata for EBS in eu-west-2
Fetched ec2 metadata for EBS in eu-west-1
[954326812311 - ap-southeast-2] Getting instances metadata...
[954326812311 - ca-central-1] Done!
[954326812311 - ap-northeast-1] Done!
[954326812311 - eu-central-1] Getting instances metadata...
[954326812311 - sa-east-1] Done!
[954326812311 - us-east-1] Getting instances metadata...
[954326812311 - ap-northeast-2] Done!
[954326812311 - us-east-2] Getting instances metadata...
[954326812311 - us-west-1] Getting instances metadata...
[954326812311 - us-west-2] Getting instances metadata...
Fetched ec2 metadata for EBS in ap-northeast-2
Fetched ec2 metadata for EBS in ap-northeast-1
Fetched ec2 metadata for EBS in sa-east-1
Fetched ec2 metadata for EBS in ca-central-1
[954326812311 - us-east-2] Done!
[954326812311 - eu-central-1] Done!
[954326812311 - us-east-1] Done!
[954326812311 - ap-southeast-2] Done!
[954326812311 - us-west-1] Done!
[954326812311 - ap-southeast-1] Done!
[954326812311 - us-west-2] Done!
Fetched ec2 metadata for EBS in ap-southeast-1
Fetched ec2 metadata for EBS in ap-southeast-2
Fetched ec2 metadata for EBS in eu-central-1
Fetched ec2 metadata for EBS in us-east-1
Fetched ec2 metadata for EBS in us-east-2
Fetched ec2 metadata for EBS in us-west-1
Fetched ec2 metadata for EBS in us-west-2
Processing billing data (1/5 - billing diff)...
Traceback (most recent call last):
File "src/get_bill_diff.py", line 65, in
first_month = preserved_months[0]
IndexError: list index out of range
Processing billing data (2/5 - instance history)...
Processing billing data (3/5 - ec2 last month)...
Processing billing data (4/5 - ebs last month)...
Processing billing data (5/5 - s3 cost)...
Generating xlsx file...
Traceback (most recent call last):
File "src/make_xlsx.py", line 630, in
main(name)
File "src/make_xlsx.py", line 612, in main
gen_weekly_variations(workbook, header_format, val_format)
File "src/make_xlsx.py", line 201, in gen_weekly_variations
with open(IN_ABSOLUTE_COST_PER_MONTH) as f:
FileNotFoundError: [Errno 2] No such file or directory: 'out/absolute.csv'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants