Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Scaling issues with web due to low CPU limits #15955

Merged
merged 3 commits into from
Sep 11, 2024
Merged

fix: Scaling issues with web due to low CPU limits #15955

merged 3 commits into from
Sep 11, 2024

Conversation

peturgq
Copy link
Member

@peturgq peturgq commented Sep 11, 2024

...

Attach a link to issue if relevant

What

Change CPU limits for WEB

Why

On startup web uses a lot of CPU, causing HPA's to spawn more and more pods

Screenshots / Gifs

Attach Screenshots / Gifs to help reviewers understand the scope of the pull request

Checklist:

  • I have performed a self-review of my own code
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • Formatting passes locally with my changes
  • I have rebased against main before asking for a review

Summary by CodeRabbit

  • New Features
    • Increased CPU limits and requests for the web service, enhancing its ability to handle workloads and potentially improving performance under load.
  • Bug Fixes
    • Adjusted resource allocation settings across development, production, and staging configurations to ensure optimal performance.

@peturgq peturgq requested a review from a team as a code owner September 11, 2024 13:23
@peturgq peturgq added the automerge Merge this PR as soon as all checks pass label Sep 11, 2024
Copy link
Contributor

coderabbitai bot commented Sep 11, 2024

Walkthrough

The changes involve updates to resource allocation settings for a web service across multiple configuration files. Specifically, CPU limits and requests have been increased from '400m' to '1000m' for limits and from '200m' to '300m' for requests. Memory settings remain unchanged. These adjustments are reflected in the web.ts, values.dev.yaml, values.prod.yaml, and values.staging.yaml files.

Changes

Files Change Summary
apps/web/infra/web.ts, charts/islandis/values.dev.yaml, charts/islandis/values.prod.yaml, charts/islandis/values.staging.yaml Increased CPU limits from '400m' to '1000m' and requests from '200m' to '300m'; memory settings unchanged.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share
Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai generate interesting stats about this repository and render them as a table.
    • @coderabbitai show all the console.log statements in this repository.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

Copy link
Member

@brynjarorng brynjarorng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

Outside diff range and nitpick comments (1)
charts/islandis/values.dev.yaml (1)

Line range hint 13-25: Consider adjusting the service fee or discount structure to better align with customer incentives.

The current implementation of a flat $20 service fee on discounted bills could negate the benefit of the discount, especially for smaller purchases or the lower loyalty tier (3-4 years at 10% discount).

For example, a $100 purchase with a 10% discount would result in a final price of $110 after adding the service fee, which is more than the undiscounted price. This might lead to customer dissatisfaction, as the intent to reward loyalty could paradoxically increase their bill.

Consider one of the following:

  1. Make the service fee a percentage of the discounted subtotal instead of a flat fee.
  2. Increase the discount percentages to account for the service fee.
  3. Only apply the service fee for discounts above a certain threshold.
  4. Waive the service fee for loyalty tiers below a certain threshold.

This will help ensure that the loyalty discount program aligns with customer expectations and incentivizes them appropriately.

Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL

Commits

Files that changed from the base of the PR and between 75e2db8 and 49c1542.

Files selected for processing (4)
  • apps/web/infra/web.ts (1 hunks)
  • charts/islandis/values.dev.yaml (1 hunks)
  • charts/islandis/values.prod.yaml (1 hunks)
  • charts/islandis/values.staging.yaml (1 hunks)
Additional context used
Path-based instructions (1)
apps/web/infra/web.ts (1)

Pattern apps/**/*: "Confirm that the code adheres to the following:

  • NextJS best practices, including file structure, API routes, and static generation methods.
  • Efficient state management and server-side rendering techniques.
  • Optimal use of TypeScript for component and utility type safety."
Additional comments not posted (5)
apps/web/infra/web.ts (1)

57-58: Approve the CPU limit and request changes, but consider the impact and monitor usage.

The increased CPU limits and requests align with the PR objective of addressing scaling issues. However, please consider the following:

  1. Monitor the web service's CPU usage to ensure the new limits are not consistently reached, which could indicate the need for further optimization or scaling.
  2. Assess the impact of these changes on the resources available for other services in the cluster.
  3. Ensure that the changes adhere to NextJS best practices for resource utilization and performance optimization.
charts/islandis/values.staging.yaml (2)

3073-3074: Verify reasoning for CPU limit increase.

The CPU limit for the web service has been significantly increased from 400m to 1000m. While this may help handle higher loads, it's important to:

  • Verify the rationale behind this 150% increase. Was it based on observed CPU usage metrics or anticipated traffic growth?
  • Ensure there is sufficient headroom in the cluster to accommodate this increase without adversely impacting other services.
  • Monitor CPU utilization and latency after applying this change to validate if it's having the intended effect.

3076-3077: LGTM!

The increase in CPU request from 200m to 300m aligns with the corresponding increase in CPU limits for the web service. This will allow the pods to start with a higher guaranteed CPU allocation.

charts/islandis/values.prod.yaml (2)

3210-3210: Verify if the increased CPU limit is necessary and optimize it based on actual utilization.

The CPU limit for the web service has been increased significantly from 400m to 1000m. While the change is approved, it's recommended to:

  • Verify if this much additional CPU capacity is truly needed to avoid overallocation of resources.
  • Monitor the CPU utilization over time and optimize the limit if the additional capacity remains underutilized.

3213-3213: LGTM!

Increasing the CPU request from 200m to 300m helps ensure the web service pods have a guaranteed minimum of 300 millicores of CPU. This reduces the risk of CPU starvation.

The request is 30% of the 1000m limit, which enables good resource utilization.

Copy link

codecov bot commented Sep 11, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 36.84%. Comparing base (75e2db8) to head (49c1542).
Report is 1 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##             main   #15955   +/-   ##
=======================================
  Coverage   36.84%   36.84%           
=======================================
  Files        6709     6709           
  Lines      137553   137553           
  Branches    39102    39102           
=======================================
  Hits        50676    50676           
  Misses      86877    86877           
Flag Coverage Δ
api 3.39% <ø> (ø)
api-domains-auth-admin 49.89% <ø> (ø)
api-domains-communications 40.54% <ø> (ø)
application-system-api 41.69% <ø> (ø)
application-template-api-modules 23.47% <ø> (-0.01%) ⬇️
cms 0.42% <ø> (ø)
cms-translations 39.70% <ø> (ø)
judicial-system-api 19.35% <ø> (ø)
judicial-system-backend 55.88% <ø> (ø)
services-user-notification 47.71% <ø> (-0.03%) ⬇️
web 1.85% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 75e2db8...49c1542. Read the comment docs.

@kodiakhq kodiakhq bot merged commit bc4f1b5 into main Sep 11, 2024
67 checks passed
@kodiakhq kodiakhq bot deleted the scaling/web branch September 11, 2024 13:56
jonnigs pushed a commit that referenced this pull request Sep 12, 2024
* fix: Scaling issues with web due to low CPU limits

* chore: charts update dirty files

---------

Co-authored-by: andes-it <builders@andes.is>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
automerge Merge this PR as soon as all checks pass
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants