Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Revert "Enable NodeKiller on ClusterLoader presubmits" #16847

Merged
merged 1 commit into from
Mar 19, 2020
Merged

Revert "Enable NodeKiller on ClusterLoader presubmits" #16847

merged 1 commit into from
Mar 19, 2020

Conversation

mm4tt
Copy link
Contributor

@mm4tt mm4tt commented Mar 19, 2020

This reverts commit 53f19ac.

Here is an example of PR when presubmit failed twice due to NK - kubernetes/perf-tests#1126
An example presubmit run https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/perf-tests/1126/pull-perf-tests-clusterloader2/1240479871608557568/
In the above run, NK killed 6 nodes during density which in the end resulted in test timing out after 1h40min.

We should change the logic to not kill the nodes so many times.

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Mar 19, 2020
@mm4tt
Copy link
Contributor Author

mm4tt commented Mar 19, 2020

/assign @jprzychodzen
/assing @mborsz

@k8s-ci-robot k8s-ci-robot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Mar 19, 2020
@k8s-ci-robot k8s-ci-robot added area/config Issues or PRs related to code in /config sig/testing Categorizes an issue or PR as relevant to SIG Testing. labels Mar 19, 2020
@k8s-ci-robot k8s-ci-robot requested review from mborsz and shyamjvs March 19, 2020 07:29
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mm4tt

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 19, 2020
@jprzychodzen
Copy link
Contributor

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 19, 2020
@oxddr
Copy link
Contributor

oxddr commented Mar 19, 2020

/hold
Could comment out those lines and add a comment with a todo? This way we won't forget to turn it on, once it's fixed.

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 19, 2020
@mm4tt
Revert "Enable NodeKiller on ClusterLoader presubmits"
0529666 
This reverts commit 53f19ac.

Here is an example of PR when presubmit failed twice due to NK - kubernetes/perf-tests#1126
An example presubmit run https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/perf-tests/1126/pull-perf-tests-clusterloader2/1240479871608557568/
In the above run, NK killed 5 nodes during density which in the end resulted in test timing out after 1h40min.
Here is the last NK log:
```
W0319 05:03:07.667] I0319 05:03:07.667437   12947 nodes.go:121] NodeKiller: Node e2e-1126-62db2-minion-group-sszg removed from killing. Runs pod prometheus-k8s-0
W0319 05:03:07.668] I0319 05:03:07.667536   12947 nodes.go:135] NodeKiller: 94 nodes available, wants to fail 1 nodes
W0319 05:03:07.668] I0319 05:03:07.667547   12947 nodes.go:140] NodeKiller: Node "e2e-1126-62db2-minion-group-ngh6" schedule for failure
W0319 05:03:07.668] I0319 05:03:07.667643   12947 nodes.go:154] NodeKiller: Stopping docker and kubelet on "e2e-1126-62db2-minion-group-ngh6" to simulate failure
```

We should change the logic to not kill the nodes so many times.
@mm4tt mm4tt force-pushed the nk_presubmit_revert branch from 7d95a61 to 0529666 Compare March 19, 2020 11:43
@k8s-ci-robot k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 19, 2020
@mm4tt
Copy link
Contributor Author

mm4tt commented Mar 19, 2020

Done, ptal
/hold cancel

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 19, 2020
@jprzychodzen
Copy link
Contributor

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 19, 2020
@oxddr
Copy link
Contributor

oxddr commented Mar 19, 2020 •
edited
Loading

Done, ptal

Thanks!

@k8s-ci-robot k8s-ci-robot merged commit 66ad706 into kubernetes:master Mar 19, 2020
@k8s-ci-robot
Copy link
Contributor

@mm4tt: Updated the job-config configmap in namespace default at cluster default using the following files:

  • key sig-scalability-presubmit-jobs.yaml using file config/jobs/kubernetes/sig-scalability/sig-scalability-presubmit-jobs.yaml

In response to this:

This reverts commit 53f19ac.

Here is an example of PR when presubmit failed twice due to NK - kubernetes/perf-tests#1126
An example presubmit run https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/perf-tests/1126/pull-perf-tests-clusterloader2/1240479871608557568/
In the above run, NK killed 6 nodes during density which in the end resulted in test timing out after 1h40min.

We should change the logic to not kill the nodes so many times.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mborsz mborsz Awaiting requested review from mborsz

@shyamjvs shyamjvs Awaiting requested review from shyamjvs

Assignees

@jprzychodzen jprzychodzen

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/config Issues or PRs related to code in /config cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. sig/testing Categorizes an issue or PR as relevant to SIG Testing. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@mm4tt @k8s-ci-robot @jprzychodzen @oxddr