Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tag Google Cloud instances with the name of the test or workflow for cost management #5153

Closed
teor2345 opened this issue Sep 13, 2022 · 8 comments · Fixed by #5693
Closed

Tag Google Cloud instances with the name of the test or workflow for cost management #5153

teor2345 opened this issue Sep 13, 2022 · 8 comments · Fixed by #5693
Labels
A-devops Area: Pipelines, CI/CD and Dockerfiles C-enhancement Category: This is an improvement I-cost Zebra infrastructure costs

Comments

@teor2345
Copy link
Contributor

teor2345 commented Sep 13, 2022 •
edited
Loading

Motivation

When we look at Google Cloud costs, we want to know which test or continuous deployment is causing them.

This is essential for reducing Google Cloud costs, which we significantly larger than our budget in September.

Priority

The priority of this ticket depends on Google Cloud costs from 24-30 September.
@teor2345 teor2345 added A-devops Area: Pipelines, CI/CD and Dockerfiles C-enhancement Category: This is an improvement S-needs-triage Status: A bug report needs triage P-Medium ⚡ I-heavy Problems with excessive memory, disk, or CPU usage I-cost Zebra infrastructure costs and removed I-heavy Problems with excessive memory, disk, or CPU usage labels Sep 13, 2022
@mpguerra mpguerra moved this to 🆕 New in Zebra Sep 22, 2022
@mpguerra mpguerra added this to Zebra Sep 22, 2022
@teor2345
Copy link
Contributor Author

I added this ticket to the sprint, because it helps us work out how to limit Google Cloud costs.

@mpguerra
Copy link
Contributor

I'm going to move this to the next sprint. Although this is important it's not high priority for this sprint.

@teor2345
Copy link
Contributor Author

I'm going to move this to the next sprint. Although this is important it's not high priority for this sprint.

This is essential for reducing Google Cloud costs, which are currently significantly larger than our budget.

So if we run out of DevOps work in this sprint, let's do this next.

@teor2345
Copy link
Contributor Author

I'm going to move this to the next sprint. Although this is important it's not high priority for this sprint.

This is essential for reducing Google Cloud costs, which are currently significantly larger than our budget.

So if we run out of DevOps work in this sprint, let's do this next.

I just looked at costs, and they have decreased significantly over the past week.

So let's check costs during sprint planning, and set the priority of this ticket then.

@teor2345
Copy link
Contributor Author

Tagging it as low for now, in the hope that costs continue to be low.

@mpguerra mpguerra removed the S-needs-triage Status: A bug report needs triage label Sep 29, 2022
@teor2345
Copy link
Contributor Author

teor2345 commented Oct 4, 2022

We only need to do this if cost is a concern.

@teor2345 teor2345 closed this as not planned Won't fix, can't repro, duplicate, stale Oct 12, 2022
Repository owner moved this from 🆕 New to ✅ Done in Zebra Oct 12, 2022
@teor2345
Copy link
Contributor Author

@mpguerra it looks like cost might be a problem, so let's re-check next week, and schedule this in the next sprint if needed?

We can get the CI failure rate down first, move jobs to the main branch, and make the send transaction test faster (#5015).
But then we need this ticket to find out which other jobs we need to speed up.

@teor2345 teor2345 reopened this Oct 13, 2022
@teor2345
Copy link
Contributor Author

teor2345 commented Nov 1, 2022

Costs were reasonable for October, so we don't need to do this unless we need to do detailed cost monitoring again.

@teor2345 teor2345 closed this as not planned Won't fix, can't repro, duplicate, stale Nov 1, 2022
@gustavovalverde gustavovalverde mentioned this issue Nov 3, 2022
Closed
@mpguerra mpguerra moved this from ✅ Done to 🛑 Won't Fix in Zebra Nov 9, 2022
gustavovalverde added a commit that referenced this issue Nov 22, 2022
@gustavovalverde
feat(gcp): add label to instances for cost and logs grouping
bb15997 
Previous behavior:
We couldn't search GCP logs using the instance name if that instance was
already deleted. And if we want to know how we're spending our budget its
also difficult to know if specific tests or type of instances are the one
responsible for a certain % of the costs

Fixes #5153
Fixses #5543

Expected behavior:
Be able to search logs using the test ID or at least the github reference,
and be able to group GCP costs by labels

Solution:
- Add labels to instances
@gustavovalverde gustavovalverde mentioned this issue Nov 22, 2022
Merged
5 tasks
@teor2345 teor2345 reopened this Nov 22, 2022
@teor2345 teor2345 changed the title Tag Google Cloud instances with the name of the test or workflow Tag Google Cloud instances with the name of the test or workflow for cost management Nov 22, 2022
@mergify mergify bot closed this as completed in #5693 Nov 24, 2022
mergify bot pushed a commit that referenced this issue Nov 24, 2022
@gustavovalverde @teor2345
feat(gcp): add label to instances for cost and logs grouping (#5693)
c774541 
* feat(gcp): add label to instances for cost and logs grouping

Previous behavior:
We couldn't search GCP logs using the instance name if that instance was
already deleted. And if we want to know how we're spending our budget its
also difficult to know if specific tests or type of instances are the one
responsible for a certain % of the costs

Fixes #5153
Fixses #5543

Expected behavior:
Be able to search logs using the test ID or at least the github reference,
and be able to group GCP costs by labels

Solution:
- Add labels to instances

* chore: add Network as a label

* Revert "chore: add Network as a label"

This reverts commit 146f747.

* Update .github/workflows/zcashd-manual-deploy.yml

Co-authored-by: teor <teor@riseup.net>

Co-authored-by: teor <teor@riseup.net>
Repository owner moved this from 🛑 Won't Fix to ✅ Done in Zebra Nov 24, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-devops Area: Pipelines, CI/CD and Dockerfiles C-enhancement Category: This is an improvement I-cost Zebra infrastructure costs
Projects
Zebra
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat(gcp): add label to instances for cost and logs grouping ZcashFoundation/zebra
2 participants
@mpguerra @teor2345