Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix memory usage when listing pods/events #1477

Merged
merged 1 commit into from
Sep 18, 2020
Merged

Fix memory usage when listing pods/events #1477

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion slo-monitor/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
# limitations under the License.

PACKAGE = k8s.io/perf-tests/slo-monitor
TAG = 0.14.0
TAG = 0.15.0
# Image should be pulled from k8s.gcr.io, which will auto-detect
# region (us, eu, asia, ...) and pull from the closest.
REPOSITORY?=k8s.gcr.io
Expand Down
33 changes: 28 additions & 5 deletions slo-monitor/src/monitors/pod_monitor.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,8 +129,20 @@ func (pm *PodStartupLatencyDataMonitor) Run(stopCh chan struct{}) error {
controller := NewWatcherWithHandler(
&cache.ListWatch{
ListFunc: func(options metav1.ListOptions) (runtime.Object, error) {
// This call is paginated by cache.Reflector.
return pm.kubeClient.CoreV1().Pods(v1.NamespaceAll).List(options)
// slo-monitor is not using the events returned from the list
// (it uses NoStoreQueue implementation of Store which discards them),
// only the resourceVersion is used to instantiate the watch from this point.
// This trick allows us to reduce memory usage on startup and further relists.
o := metav1.ListOptions{
wojtek-t marked this conversation as resolved.
Show resolved Hide resolved
Limit: 1,
}
result, err := pm.kubeClient.CoreV1().Pods(v1.NamespaceAll).List(o)
if err != nil {
return nil, err
}
result.Continue = ""
wojtek-t marked this conversation as resolved.
Show resolved Hide resolved
result.Items = nil
return result, nil
},
WatchFunc: func(options metav1.ListOptions) (watch.Interface, error) {
return pm.kubeClient.CoreV1().Pods(v1.NamespaceAll).Watch(options)
Expand Down Expand Up @@ -175,9 +187,20 @@ func (pm *PodStartupLatencyDataMonitor) Run(stopCh chan struct{}) error {
eventController := NewWatcherWithHandler(
&cache.ListWatch{
ListFunc: func(options metav1.ListOptions) (runtime.Object, error) {
options.FieldSelector = eventSelector
// This call is paginated by cache.Reflector.
return pm.kubeClient.CoreV1().Events(v1.NamespaceAll).List(options)
// slo-monitor is not using the pods returned from the list
// (it uses NoStoreQueue implementation of Store which discards them),
// only the resourceVersion is used to instantiate the watch from this point.
// This trick allows us to reduce memory usage on startup and further relists.
o := metav1.ListOptions{
wojtek-t marked this conversation as resolved.
Show resolved Hide resolved
Limit: 1,
}
result, err := pm.kubeClient.CoreV1().Events(v1.NamespaceAll).List(o)
if err != nil {
return nil, err
}
result.Continue = ""
result.Items = nil
return result, nil
},
WatchFunc: func(options metav1.ListOptions) (watch.Interface, error) {
options.FieldSelector = eventSelector
Expand Down
2 changes: 0 additions & 2 deletions slo-monitor/src/monitors/watcher.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,6 @@ import (

const (
resyncPeriod = time.Duration(0)
watchListPageSize = 5000
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mborsz - given we don't need this, do we want to revert the changes you were doing?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, once we validate this approach, we can submit follow up PR that will remove this, fork of client-go and other ugly stuff I added.

)

// NewWatcherWithHandler creates a simple watcher that will call `h` for all coming objects
Expand All @@ -41,7 +40,6 @@ func NewWatcherWithHandler(lw cache.ListerWatcher, objType runtime.Object, setHa
ObjectType: objType,
FullResyncPeriod: resyncPeriod,
RetryOnError: false,
WatchListPageSize: watchListPageSize,

Process: func(obj interface{}) error {
workItem := obj.(workItem)
Expand Down