Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Log parent quota #8390

Merged
merged 1 commit into from
Feb 22, 2022
Merged

Log parent quota #8390

merged 1 commit into from
Feb 22, 2022

Conversation

Furisto
Copy link
Member

@Furisto Furisto commented Feb 22, 2022

Description

Log the quota of the parent. Today with https://github.com/gitpod-io/ops/issues/1244 and in yesterdays incident we saw that the parent value can be the reason for not being able to set cgroup values. It should therefore be included in the error to speed up analysis.

Related Issue(s)

n.a.

How to test

  • Edit ws-daemon config-map
 "cpulimit": {
          "enabled": true,
          "totalBandwidth": "12",
          "limit": "2",
          "burstLimit": "6",
          "controlPeriod": "15s",
          "cgroupBasePath": "/mnt/node-cgroups"
        },
  • Edit ws-manager config map
  "limits": {
              "cpu": "4",
              "memory": "",
              "ephemeral-storage": ""
            }
  • Restart ws-daemon and ws-manager
  • Open workspace and run stress in the workspace

Release Notes

NONE

@Furisto Furisto requested a review from a team February 22, 2022 15:17
@github-actions github-actions bot added the team: workspace Issue belongs to the Workspace team label Feb 22, 2022
@kylos101
Copy link
Contributor

@Furisto may I ask you to share how you tested? I'm not sure if this is something you can do in core-dev, or workspace-preview?

@Furisto
Copy link
Member Author

Furisto commented Feb 22, 2022

@kylos101 I think you could overwrite the configmap of the ws-manager. Let me check...

@Furisto
Copy link
Member Author

Furisto commented Feb 22, 2022

{"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","error":"cannot set CFS quota of 600000 (period is 100000, parent quota is 400000):\n github.com/gitpod-io/gitpod/ws-daemon/pkg/cpulimit.CgroupCFSController.SetLimit\n github.com/gitpod-io/gitpod/ws-daemon/pkg/cpulimit/cfs.go:55\n - write /mnt/node-cgroups/cpu/kubepods/burstable/pod3c3aed9f-859c-4880-a5ee-3a228c197dc3/bfb29396e078697c68937e6ef81a64763b6bad965c3b75f3b565c100a7b8103c/cpu.cfs_quota_us: invalid argument","instanceId":"91cfb18d-2c2a-4cae-afa9-e44710eef399","level":"warning","message":"cannot set CPU limit","serviceContext":{"service":"ws-daemon","version":"commit-4c27eb3eb8d2689ee94aae2529078e131bfd55f2"},"severity":"WARNING","time":"2022-02-22T17:24:58Z","userId":"","workspaceId":"gitpodio-templatetypescr-zgeoncnwteu"}

@Furisto
Copy link
Member Author

Furisto commented Feb 22, 2022

@kylos Updated test instructions

@roboquat roboquat merged commit 7ca6184 into main Feb 22, 2022
@roboquat roboquat deleted the fo/p-quota branch February 22, 2022 18:16
@roboquat roboquat added deployed: workspace Workspace team change is running in production deployed Change is completely running in production labels Feb 24, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
deployed: workspace Workspace team change is running in production deployed Change is completely running in production release-note-none size/S team: workspace Issue belongs to the Workspace team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants