-
Notifications
You must be signed in to change notification settings - Fork 43
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Support resource configuration for imagebuilder (#351)
<!-- Thanks for sending a pull request! Here are some tips for you: 1. Run unit tests and ensure that they are passing 2. If your change introduces any API changes, make sure to update the e2e tests 3. Make sure documentation is updated for your PR! --> **What this PR does / why we need it**: <!-- Explain here the context and why you're making the change. What is the problem you're trying to solve. ---> Currently, Merlin image building jobs don't support resource configuration, and requests.cpu is hardcoded. Without requests.memory configured on them, it's common to see jobs failing with OOMKilled error if multiple jobs are running concurrently. This PR supports resource configuration so that some reasonable amount of memory can be set to mitigate this issue. In addition, the GKE autoscaler can spin up another node if multiple jobs are running concurrently, instead of placing all of the jobs onto the same node, and causing them to fight for available memory. **Which issue(s) this PR fixes**: <!-- *Automatically closes linked issue when PR is merged. Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`. --> Fixes # **Does this PR introduce a user-facing change?**: <!-- If no, just write "NONE" in the release-note block below. If yes, a release note is required. Enter your extended release note in the block below. If the PR requires additional action from users switching to the new release, include the string "action required". For more information about release notes, see kubernetes' guide here: http://git.k8s.io/community/contributors/guide/release-notes.md --> ```release-note NONE ``` **Checklist** - [x] Added unit test, integration, and/or e2e tests
- Loading branch information
1 parent
14d23c9
commit 101989d
Showing
12 changed files
with
207 additions
and
47 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,48 @@ | ||
| // Copyright 2020 The Merlin Authors | ||
| // | ||
| // Licensed under the Apache License, Version 2.0 (the "License"); | ||
| // you may not use this file except in compliance with the License. | ||
| // You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, software | ||
| // distributed under the License is distributed on an "AS IS" BASIS, | ||
| // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| // See the License for the specific language governing permissions and | ||
| // limitations under the License. | ||
|
|
||
| package imagebuilder | ||
|
|
||
| import ( | ||
| corev1 "k8s.io/api/core/v1" | ||
| "k8s.io/apimachinery/pkg/api/resource" | ||
| ) | ||
|
|
||
| // RequestLimitResources is a Kubernetes resource request and limits | ||
| type RequestLimitResources struct { | ||
| Request Resource | ||
| Limit Resource | ||
| } | ||
|
|
||
| // Build converts the spec into a Kubernetes spec | ||
| func (r *RequestLimitResources) Build() corev1.ResourceRequirements { | ||
| return corev1.ResourceRequirements{ | ||
| Requests: r.Request.Build(), | ||
| Limits: r.Limit.Build(), | ||
| } | ||
| } | ||
|
|
||
| // Resource is a Kubernetes resource | ||
| type Resource struct { | ||
| CPU resource.Quantity | ||
| Memory resource.Quantity | ||
| } | ||
|
|
||
| // Build converts the spec into a Kubernetes spec | ||
| func (r *Resource) Build() corev1.ResourceList { | ||
| return corev1.ResourceList{ | ||
| corev1.ResourceCPU: r.CPU, | ||
| corev1.ResourceMemory: r.Memory, | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,57 @@ | ||
| // Copyright 2020 The Merlin Authors | ||
| // | ||
| // Licensed under the Apache License, Version 2.0 (the "License"); | ||
| // you may not use this file except in compliance with the License. | ||
| // You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, software | ||
| // distributed under the License is distributed on an "AS IS" BASIS, | ||
| // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| // See the License for the specific language governing permissions and | ||
| // limitations under the License. | ||
|
|
||
| package imagebuilder | ||
|
|
||
| import ( | ||
| "testing" | ||
|
|
||
| "github.com/stretchr/testify/assert" | ||
| corev1 "k8s.io/api/core/v1" | ||
| "k8s.io/apimachinery/pkg/api/resource" | ||
| ) | ||
|
|
||
| var ( | ||
| cpu = resource.MustParse("500m") | ||
| memory = resource.MustParse("500Mi") | ||
| ) | ||
|
|
||
| func CreateRequestLimitResources() RequestLimitResources { | ||
| return RequestLimitResources{ | ||
| Request: Resource{ | ||
| CPU: cpu, | ||
| Memory: memory, | ||
| }, | ||
| Limit: Resource{ | ||
| CPU: cpu, | ||
| Memory: memory, | ||
| }, | ||
| } | ||
| } | ||
|
|
||
| func TestContainer(t *testing.T) { | ||
| expected := corev1.ResourceRequirements{ | ||
| Requests: corev1.ResourceList{ | ||
| corev1.ResourceCPU: cpu, | ||
| corev1.ResourceMemory: memory, | ||
| }, | ||
| Limits: corev1.ResourceList{ | ||
| corev1.ResourceCPU: cpu, | ||
| corev1.ResourceMemory: memory, | ||
| }, | ||
| } | ||
| c := CreateRequestLimitResources() | ||
|
|
||
| assert.Equal(t, expected, c.Build()) | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,4 @@ | ||
| apiVersion: v1 | ||
| description: Kubernetes-friendly ML model management, deployment, and serving. | ||
| name: merlin | ||
| version: 0.8.1 | ||
| version: 0.8.2 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.