Add commands to generate cost of each GCP project #1947

yuvipanda · 2022-11-23T10:06:17Z

Fixes #1921

Things left to do:

Document, step by step, how to ensure that every new project has proper billing info set up
Document how the generator script can be run
Document who should have access to this google sheet

github-actions · 2022-11-24T00:17:17Z

Merging this PR will trigger the following deployment actions.

Support and Staging deployments

Cloud Provider	Cluster Name	Upgrade Support?	Upgrade Staging?	Reason for Staging Redeploy
gcp	callysto	No	Yes	Core infrastructure has been modified
gcp	2i2c	No	Yes	Core infrastructure has been modified
gcp	cloudbank	No	Yes	Core infrastructure has been modified
aws	2i2c-aws-us	No	Yes	Core infrastructure has been modified
aws	openscapes	No	Yes	Core infrastructure has been modified
aws	nasa-veda	No	Yes	Core infrastructure has been modified
aws	gridsst	No	Yes	Core infrastructure has been modified
gcp	pangeo-hubs	No	Yes	Core infrastructure has been modified
gcp	linked-earth	No	Yes	Core infrastructure has been modified
gcp	meom-ige	No	Yes	Core infrastructure has been modified
kubeconfig	utoronto	No	Yes	Core infrastructure has been modified
aws	carbonplan	No	Yes	Core infrastructure has been modified
gcp	m2lines	No	Yes	Core infrastructure has been modified
aws	victor	No	Yes	Core infrastructure has been modified
gcp	leap	No	Yes	Core infrastructure has been modified
gcp	awi-ciroh	No	Yes	Core infrastructure has been modified
gcp	qcl	No	Yes	Core infrastructure has been modified
gcp	2i2c-uk	No	Yes	Core infrastructure has been modified
aws	ubc-eoas	No	Yes	Core infrastructure has been modified
aws	nasa-cryo	No	Yes	Core infrastructure has been modified

Production deployments

Cloud Provider	Cluster Name	Hub Name	Reason for Redeploy
gcp	callysto	prod	Core infrastructure has been modified
gcp	2i2c	demo	Core infrastructure has been modified
gcp	2i2c	ohw	Core infrastructure has been modified
gcp	2i2c	pfw	Core infrastructure has been modified
gcp	2i2c	peddie	Core infrastructure has been modified
gcp	2i2c	catalyst-cooperative	Core infrastructure has been modified
gcp	2i2c	earthlab	Core infrastructure has been modified
gcp	2i2c	paleohack2021	Core infrastructure has been modified
gcp	2i2c	aup	Core infrastructure has been modified
gcp	2i2c	temple	Core infrastructure has been modified
gcp	2i2c	ucmerced	Core infrastructure has been modified
gcp	2i2c	cosmicds	Core infrastructure has been modified
gcp	cloudbank	ccsf	Core infrastructure has been modified
gcp	cloudbank	csm	Core infrastructure has been modified
gcp	cloudbank	elcamino	Core infrastructure has been modified
gcp	cloudbank	glendale	Core infrastructure has been modified
gcp	cloudbank	howard	Core infrastructure has been modified
gcp	cloudbank	miracosta	Core infrastructure has been modified
gcp	cloudbank	skyline	Core infrastructure has been modified
gcp	cloudbank	demo	Core infrastructure has been modified
gcp	cloudbank	fresno	Core infrastructure has been modified
gcp	cloudbank	laney	Core infrastructure has been modified
gcp	cloudbank	lassen	Core infrastructure has been modified
gcp	cloudbank	sbcc	Core infrastructure has been modified
gcp	cloudbank	lacc	Core infrastructure has been modified
gcp	cloudbank	mills	Core infrastructure has been modified
gcp	cloudbank	palomar	Core infrastructure has been modified
gcp	cloudbank	pasadena	Core infrastructure has been modified
gcp	cloudbank	sjcc	Core infrastructure has been modified
gcp	cloudbank	tuskegee	Core infrastructure has been modified
gcp	cloudbank	avc	Core infrastructure has been modified
gcp	cloudbank	csu	Core infrastructure has been modified
aws	2i2c-aws-us	researchdelight	Core infrastructure has been modified
aws	openscapes	prod	Core infrastructure has been modified
aws	nasa-veda	prod	Core infrastructure has been modified
aws	gridsst	prod	Core infrastructure has been modified
gcp	pangeo-hubs	prod	Core infrastructure has been modified
gcp	linked-earth	prod	Core infrastructure has been modified
gcp	meom-ige	prod	Core infrastructure has been modified
gcp	meom-ige	drakkar-demo	Core infrastructure has been modified
kubeconfig	utoronto	prod	Core infrastructure has been modified
kubeconfig	utoronto	r-prod	Core infrastructure has been modified
aws	carbonplan	prod	Core infrastructure has been modified
gcp	m2lines	prod	Core infrastructure has been modified
aws	victor	prod	Core infrastructure has been modified
gcp	leap	prod	Core infrastructure has been modified
gcp	awi-ciroh	prod	Core infrastructure has been modified
gcp	qcl	prod	Core infrastructure has been modified
gcp	2i2c-uk	lis	Core infrastructure has been modified
aws	ubc-eoas	prod	Core infrastructure has been modified
aws	nasa-cryo	prod	Core infrastructure has been modified

Fixes 2i2c-org#1921

pnasrat

I did a first pass and I've a few structural questions around this change in terms of where config should live, and more minor suggestions

deployer/billing.py

config/billing-accounts.yaml

deployer/billing.py

for more information, see https://pre-commit.ci

This is confusing and not actually helpful for billing

for more information, see https://pre-commit.ci

This is done via JSON Schema now

docs/hub-deployment-guide/cloud-accounts/new-gcp-project.md

yuvipanda · 2023-03-30T11:46:16Z

I want to open an issue for tracking the 'single source of truth' about our contracts and which ones are cloud cost pass through, but I see it's already a line item in #2431

deployer/billing.py

pnasrat

Two minor comments but basically looks good, this is looking good and I've already been testing out the functionality.

pnasrat · 2023-03-30T12:08:40Z

deployer/billing.py

+    """
+    match = re.match(r"(\d\d\d\d)-(\d\d)", month_str)
+    if not match:
+        raise typer.BadParameter(f"{month_str} should be formatted as YYYY-MM")


give an example should be formatted as YYYY-MM e.g. 2022-01

pnasrat

lgtm

yuvipanda · 2023-03-30T19:14:25Z

yay!

github-actions · 2023-03-30T19:26:36Z

🎉🎉🎉🎉

Monitor the deployment of the hubs here 👉 https://github.com/2i2c-org/infrastructure/actions/runs/4568225181

yuvipanda · 2023-03-30T19:30:45Z

yay thank you for your review and encouragement in getting this done, @pnasrat!

yuvipanda mentioned this pull request Nov 23, 2022

[Billing] Automatically figure out how much a dedicated GCP project spends in a month #1921

Closed

4 tasks

damianavila assigned yuvipanda Nov 23, 2022

yuvipanda added 7 commits January 17, 2023 14:32

Add commands to generate cost of each GCP project

41a46d8

Fixes 2i2c-org#1921

Add packages used for billing info

584e893

Write section separator at correct place

a4112f8

Declutter by not displaying 0$ non-charges

88dbac4

Add ability to specify time period for billing data

7c11f5f

Allow csv output

21f0f6a

Add ability to post GCP cost table to Google Sheets

239c6e6

yuvipanda force-pushed the billing-gcp branch from 1e0c54f to 239c6e6 Compare January 17, 2023 22:32

Write a documentation

670ddb6

damianavila requested a review from a team February 1, 2023 17:00

damianavila mentioned this pull request Feb 15, 2023

Expose cloud billing dashboards for customers with dedicated clusters #1575

Closed

3 tasks

damianavila assigned pnasrat Feb 15, 2023

pnasrat suggested changes Feb 16, 2023

View reviewed changes

deployer/billing.py Show resolved Hide resolved

deployer/billing.py Outdated Show resolved Hide resolved

config/billing-accounts.yaml Outdated Show resolved Hide resolved

deployer/billing.py Show resolved Hide resolved

deployer/billing.py Outdated Show resolved Hide resolved

damianavila unassigned yuvipanda Mar 1, 2023

pnasrat and others added 7 commits March 16, 2023 11:22

Merge branch 'master' into billing-gcp

16f0a7f

[pre-commit.ci] auto fixes from pre-commit.com hooks

ef97dd0

for more information, see https://pre-commit.ci

Move billing info to cluster.yaml

9e00a80

Merge remote-tracking branch 'upstream/master' into billing-gcp

54b9a7b

Don't repeat projects

0a70a8d

Don't display 'total without credits'

c4abff8

This is confusing and not actually helpful for billing

Show 6 months of costs by default

0daeed5

yuvipanda force-pushed the billing-gcp branch from b439dfe to 0daeed5 Compare March 28, 2023 08:26

pre-commit-ci bot and others added 5 commits March 28, 2023 08:26

[pre-commit.ci] auto fixes from pre-commit.com hooks

d8f00aa

for more information, see https://pre-commit.ci

Remove unused import

c108f0c

Add note about spliiting output methods

01a01be

Fix typo in cluster.yaml schema

fe9d5a0

Enforce bigquery config being present if we pay for the project

02d2d6c

Remove unneeded validate-billing-export

66a8301

This is done via JSON Schema now

pnasrat reviewed Mar 28, 2023

View reviewed changes

docs/hub-deployment-guide/cloud-accounts/new-gcp-project.md Outdated Show resolved Hide resolved

damianavila assigned yuvipanda Mar 29, 2023

yuvipanda added 4 commits March 30, 2023 15:35

Add information on how to create billing export

cea87a8

Link storage layer appropriately

c601b6c

Add docs on running the billing script

3158d6c

Add info on how to find billing info

fe96eb9

yuvipanda marked this pull request as ready for review March 30, 2023 11:42

yuvipanda mentioned this pull request Mar 30, 2023

Automate setting up bigquery billing export for new billing accounts #2442

Closed

Merge remote-tracking branch 'upstream/master' into billing-gcp

3452438

yuvipanda force-pushed the billing-gcp branch from f09569b to 3452438 Compare March 30, 2023 11:49

pnasrat reviewed Mar 30, 2023

View reviewed changes

deployer/billing.py Outdated Show resolved Hide resolved

pnasrat suggested changes Mar 30, 2023

View reviewed changes

yuvipanda added 2 commits March 31, 2023 00:34

Remove outdated TODOs

f678771

Provide example of date formatting

70e7959

pnasrat approved these changes Mar 30, 2023

View reviewed changes

Merge branch 'master' into billing-gcp

930618c

yuvipanda merged commit bc7eb18 into 2i2c-org:master Mar 30, 2023

damianavila unassigned pnasrat Apr 10, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add commands to generate cost of each GCP project #1947

Add commands to generate cost of each GCP project #1947

yuvipanda commented Nov 23, 2022 •

edited

Loading

github-actions bot commented Nov 24, 2022 •

edited

Loading

pnasrat left a comment

yuvipanda commented Mar 30, 2023

pnasrat left a comment

pnasrat Mar 30, 2023

yuvipanda Mar 30, 2023

pnasrat left a comment

yuvipanda commented Mar 30, 2023

github-actions bot commented Mar 30, 2023

yuvipanda commented Mar 30, 2023

Add commands to generate cost of each GCP project #1947

Add commands to generate cost of each GCP project #1947

Conversation

yuvipanda commented Nov 23, 2022 • edited Loading

github-actions bot commented Nov 24, 2022 • edited Loading

Support and Staging deployments

Production deployments

pnasrat left a comment

Choose a reason for hiding this comment

yuvipanda commented Mar 30, 2023

pnasrat left a comment

Choose a reason for hiding this comment

pnasrat Mar 30, 2023

Choose a reason for hiding this comment

yuvipanda Mar 30, 2023

Choose a reason for hiding this comment

pnasrat left a comment

Choose a reason for hiding this comment

yuvipanda commented Mar 30, 2023

github-actions bot commented Mar 30, 2023

yuvipanda commented Mar 30, 2023

yuvipanda commented Nov 23, 2022 •

edited

Loading

github-actions bot commented Nov 24, 2022 •

edited

Loading