Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

OOMKilled when starting an operator #579

Closed
sujaymansingh opened this issue May 21, 2019 · 2 comments
Closed

OOMKilled when starting an operator #579

sujaymansingh opened this issue May 21, 2019 · 2 comments

Comments

@sujaymansingh
Copy link
Contributor

I’ve tried installing seldon to a k8s cluster. I ran (as per the docs)

  • helm install seldon-core-operator --name seldon-core-sujay --repo https://storage.googleapis.com/seldon-charts --set usage_metrics.enabled=true --namespace sujay to install the core operator
  • helm install stable/ambassador --name ambassador-sujay --namespace sujay --set image.tag=0.40.2

Then I deployed the sample seldon chart at https://docs.seldon.io/projects/seldon-core/en/latest/graph/inference-graph.html

I have a seldon deployment:

$ kubectl get sdep
NAME           AGE
seldon-model   20m

The operator kept being killed by the cluster however.

$ kubectl get pods --namespace sujay seldon-operator-controller-manager-0
seldon-operator-controller-manager-0           0/1     OOMKilled          1          59s

The memory limits come from the helm chart itself.

        resources:
          limits:
            cpu: 100m
            memory: 30Mi
          requests:
            cpu: 100m
            memory: 20Mi

Either more memory needs to be assigned, or the operator needs to be changed so that there is no chance of it taking up that much memory.
@ryandawsonuk
Copy link
Contributor

This just recently got changed - 584d469

I tried helm install seldon-core-operator --name seldon-core-sujay --repo https://storage.googleapis.com/seldon-charts --set usage_metrics.enabled=true --namespace sujay --dry-run --debug and searched the output and the limits are no longer there. So you should be able to try again now.

@ukclivecox
Copy link
Contributor

Fixed in master

agrski added a commit that referenced this issue Dec 2, 2022
@agrski
Override config file with CLI args for Seldon CLI (#579)
3293394 
* Rename command vars for subcommands for concision

* Use consistent var assignment syntax for top-level CLI commands

* Rename private functions for clarity

Using terse contractions of terms hurts legibility in composite names, as for functions.

* Use var for command flags instead of calling methods for every flag access & modify

* Pass scheduler-host-is-set indicator to scheduler client constructor

This allows us to determine the relative priorities of config from different places.

* Do not override scheduler host explicit CLI arg with config file value

Instead, only override default args not set explicitly when running the command.

* Pass indicator for infer host being set explicitly to inference client constructor

* Do not override infer host explicit CLI arg with config file value

* Do not override Kafka broker host explicit CLI arg with config file value
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@sujaymansingh @ukclivecox @ryandawsonuk