Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Production - [Alerting] Helix API Average Response Time #4818

Closed
dotnet-eng-status bot opened this issue Jan 22, 2025 · 24 comments
Closed

Production - [Alerting] Helix API Average Response Time #4818

dotnet-eng-status bot opened this issue Jan 22, 2025 · 24 comments
Assignees
@meghnave
Labels
Critical Grafana Alert Issues opened by Grafana Inactive Alert Issues from Grafana alerts that are now "OK" Ops - First Responder Production Tied to the Production environment (as opposed to Staging)

Comments

@dotnet-eng-status
Copy link

💔 Metric state changed to alerting

Helix API Average Response Time is high!

  • Server response time 9807.247868353154

Go to rule

@dotnet/dnceng, @dotnet/prodconsvcs, please investigate

Automation information below, do not change

Grafana-Automated-Alert-Id-24cae10d9eca44079e7cf3d47f148497
@dotnet-eng-status dotnet-eng-status bot added Active Alert Issues from Grafana alerts that are now active Critical Grafana Alert Issues opened by Grafana Ops - First Responder Production Tied to the Production environment (as opposed to Staging) labels Jan 22, 2025
@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💔 Metric state changed to alerting

Helix API Average Response Time is high!

  • Server response time 9872.927123720636

Go to rule

@dotnet-eng-status dotnet-eng-status bot added Inactive Alert Issues from Grafana alerts that are now "OK" and removed Active Alert Issues from Grafana alerts that are now active labels Jan 22, 2025
@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💚 Metric state changed to ok

Helix API Average Response Time is high!

Go to rule

@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💚 Metric state changed to ok

Helix API Average Response Time is high!

Go to rule

@dotnet-eng-status dotnet-eng-status bot removed the Inactive Alert Issues from Grafana alerts that are now "OK" label Jan 23, 2025
@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💔 Metric state changed to alerting

Helix API Average Response Time is high!

  • Server response time 5759.063019502978

Go to rule

@dotnet-eng-status dotnet-eng-status bot added the Active Alert Issues from Grafana alerts that are now active label Jan 23, 2025
@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💔 Metric state changed to alerting

Helix API Average Response Time is high!

  • Server response time 6066.613860121077

Go to rule

@dotnet-eng-status dotnet-eng-status bot removed the Active Alert Issues from Grafana alerts that are now active label Jan 23, 2025
@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💚 Metric state changed to ok

Helix API Average Response Time is high!

Go to rule

@dotnet-eng-status dotnet-eng-status bot added the Inactive Alert Issues from Grafana alerts that are now "OK" label Jan 23, 2025
@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💚 Metric state changed to ok

Helix API Average Response Time is high!

Go to rule

@dotnet-eng-status dotnet-eng-status bot added Active Alert Issues from Grafana alerts that are now active and removed Inactive Alert Issues from Grafana alerts that are now "OK" labels Jan 23, 2025
@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💔 Metric state changed to alerting

Helix API Average Response Time is high!

  • Server response time 8198.871148297236

Go to rule

@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💔 Metric state changed to alerting

Helix API Average Response Time is high!

  • Server response time 8361.744495217708

Go to rule

@dotnet-eng-status dotnet-eng-status bot added Inactive Alert Issues from Grafana alerts that are now "OK" and removed Active Alert Issues from Grafana alerts that are now active labels Jan 23, 2025
@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💚 Metric state changed to ok

Helix API Average Response Time is high!

Go to rule

@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💚 Metric state changed to ok

Helix API Average Response Time is high!

Go to rule

@dotnet-eng-status dotnet-eng-status bot added Active Alert Issues from Grafana alerts that are now active and removed Inactive Alert Issues from Grafana alerts that are now "OK" labels Jan 23, 2025
@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💔 Metric state changed to alerting

Helix API Average Response Time is high!

  • Server response time 6911.773031626326

Go to rule

@dotnet-eng-status dotnet-eng-status bot removed the Active Alert Issues from Grafana alerts that are now active label Jan 24, 2025
@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💚 Metric state changed to ok

Helix API Average Response Time is high!

Go to rule

@dotnet-eng-status dotnet-eng-status bot added Inactive Alert Issues from Grafana alerts that are now "OK" Active Alert Issues from Grafana alerts that are now active and removed Inactive Alert Issues from Grafana alerts that are now "OK" labels Jan 27, 2025
@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💔 Metric state changed to alerting

Helix API Average Response Time is high!

  • Server response time 7294.108970197067

Go to rule

@dotnet-eng-status dotnet-eng-status bot added Inactive Alert Issues from Grafana alerts that are now "OK" and removed Active Alert Issues from Grafana alerts that are now active labels Jan 28, 2025
@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💚 Metric state changed to ok

Helix API Average Response Time is high!

Go to rule

@dotnet-eng-status dotnet-eng-status bot added Active Alert Issues from Grafana alerts that are now active and removed Inactive Alert Issues from Grafana alerts that are now "OK" labels Jan 28, 2025
@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💔 Metric state changed to alerting

Helix API Average Response Time is high!

  • Server response time 13772.852212847103

Go to rule

@dotnet-eng-status dotnet-eng-status bot removed the Active Alert Issues from Grafana alerts that are now active label Jan 28, 2025
@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💚 Metric state changed to ok

Helix API Average Response Time is high!

Go to rule

@dotnet-eng-status dotnet-eng-status bot added Inactive Alert Issues from Grafana alerts that are now "OK" Active Alert Issues from Grafana alerts that are now active and removed Inactive Alert Issues from Grafana alerts that are now "OK" labels Jan 28, 2025
@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💔 Metric state changed to alerting

Helix API Average Response Time is high!

  • Server response time 9786.400097753922

Go to rule

@dotnet-eng-status dotnet-eng-status bot removed the Active Alert Issues from Grafana alerts that are now active label Jan 28, 2025
@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💚 Metric state changed to ok

Helix API Average Response Time is high!

Go to rule

@dotnet-eng-status dotnet-eng-status bot added the Inactive Alert Issues from Grafana alerts that are now "OK" label Jan 28, 2025
@meghnave meghnave self-assigned this Jan 31, 2025
@meghnave
Copy link

This comment by Stu is still essentially true where in GET /swagger/ui/swagger-ui-bundle.js and QuotaManagerActorService/ReceiveReminderAsync are taking a long time with no obvious system issues.

This was referenced Feb 19, 2025
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@meghnave meghnave

Labels
Critical Grafana Alert Issues opened by Grafana Inactive Alert Issues from Grafana alerts that are now "OK" Ops - First Responder Production Tied to the Production environment (as opposed to Staging)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@meghnave