Skip to content
This repository has been archived by the owner on Jan 8, 2024. It is now read-only.

Remote runner job for Nomad clusters OOMs when doing basic tasks #1302

Closed
nateberkopec opened this issue Apr 8, 2021 · 4 comments · Fixed by #1318
Closed

Remote runner job for Nomad clusters OOMs when doing basic tasks #1302

nateberkopec opened this issue Apr 8, 2021 · 4 comments · Fixed by #1318
Assignees
@briancain
Labels
bug Something isn't working core/runner enhancement New feature or request plugin/nomad
Milestone
0.3.x

Comments

@nateberkopec
Copy link

Pretty simple: I tried using the new runner created by waypoint install as a remote runner, pulling a git repo. The runner got OOM-killed before it even finished pulling the repo down.

Not sure what the solution is - configurable resource limits? Maybe just generate some nomad jobspecs that I can modify with resource stanzas appropriate for my setup?
@briancain
Copy link
Member

Hey there @nateberkopec ! Can you provide us with the waypoint runner logs when it crashes? Were you pulling a git repo through a project created in the UI or the CLI? Thanks!

@briancain briancain added bug Something isn't working core/runner plugin/nomad labels Apr 9, 2021
@briancain briancain added this to the 0.3.x milestone Apr 9, 2021
@mitchellh
Copy link
Contributor

@briancain I think the issue is that our default runner resource requests are too low. I don't see us explicitly setting any so we're probably inheriting some low defaults from somewhere. We should increase them.

@briancain briancain added the enhancement New feature or request label Apr 9, 2021
@mitchellh
Copy link
Contributor

Working on unrelated Nomad things, I grabbed a screenshot. These are some defaults from somewhere and are definitely too low.

image

@nateberkopec
Copy link
Author

Yup. And then I just see some messages regarding OOM killing in the Nomad job log once I actually try to waypoint up -remote.

@briancain briancain self-assigned this Apr 12, 2021
briancain added a commit that referenced this issue Apr 12, 2021
@briancain
serverinstall/nomad: Provide config values for memory and CPU
3c2e027 
This commit introduces config options to use when defining a Server and
Runner nomad job config. If no option is specified, it will use the
configured defaults for Waypoint. This commit also increases the default
CPU in MHz to 200 and Memory in MB to 600 (double the default for Nomad
tasks)

Fixes #1302
@briancain briancain mentioned this issue Apr 12, 2021
Merged
briancain added a commit that referenced this issue Apr 13, 2021
@briancain
serverinstall/nomad: Provide config values for memory and CPU
f8407b5 
This commit introduces config options to use when defining a Server and
Runner nomad job config. If no option is specified, it will use the
configured defaults for Waypoint. This commit also increases the default
CPU in MHz to 200 and Memory in MB to 600 (double the default for Nomad
tasks)

Fixes #1302
@hc-waypoint-bot hc-waypoint-bot mentioned this issue Apr 20, 2021
Merged
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@briancain briancain

Labels
bug Something isn't working core/runner enhancement New feature or request plugin/nomad
Projects
None yet
Milestone
0.3.x
Development

Successfully merging a pull request may close this issue.

{serverinstall/builtin}/nomad: Provide config values for memory and CPU hashicorp/waypoint
3 participants
@mitchellh @briancain @nateberkopec