zoekt-webserver regularly using 50GB memory, OOMing at 60GB

[emidoots]

On an xlarge customer deployment (https://app.hubspot.com/contacts/2762526/company/407948923) I am observing zoekt-webserver regularly using ~50GB memory and sometimes OOMing when it goes over 60GB which is the limit we have in place. We've mitigated by bumping up to 80GB but it seems a little crazy to have to do this.

Is there a way we can tell Zoekt "you can use N GB memory, but after that slow down a bit so you don't OOM"? Or is there any low-hanging fruit w.r.t. zoekt-webserver that would reduce memory consumption?

cc @sourcegraph/code-search @sourcegraph/core-services

[keegancsmith]

Maybe you can try and increase the shard size flag? For a large repository I believe it may result in overall smaller indexes and lower memory usage in a request. Another possibility is reducing the number of results found per shard.

I don't think there is low hanging fruit other than what I mentioned, but maybe an inspection of some memory traces would reveal some. Otherwise there are some likely higher level changes which can be made:

• Make indexes mmaped as well.
• Horizontally scale zoekt-webserver.
• Smarter decisions in which shards to search and in what order.

[emidoots]

@keegancsmith I briefly looked at CLI flags for zoekt-webserver and zoekt-indexserver but didn't find a shard flag size. Can you point me to it? Also, what would you recommend trying to set it to? If we alter it how would it affect the existing indexes produced by Zoekt? (would it need to re-index or something)

[keegancsmith]

@keegancsmith I briefly looked at CLI flags for zoekt-webserver and zoekt-indexserver but didn't find a shard flag size. Can you point me to it?

-shard_limit https://sourcegraph.com/github.com/sourcegraph/zoekt/-/blob/cmd/flags.go#L42:26
It would be for the index server. You would probably have to update our indexserver to set the field, we justrely on the default. @ijsnow recently added functionality allowing us to pass flags from sourcegraph down to the indexserver, so that may be the best place to do it.

Also, what would you recommend trying to set it to?

Zoekt divides its parallelism up by shards (ie there is one goroutine per shard). So how long it takes to search a shard is the critical path for how quick zoekt is assuming you had infinite cores. 100mb is quite small. As an educated guess I would suspect 1G would still result in fast searches.

If we alter it how would it affect the existing indexes produced by Zoekt? (would it need to re-index or something)

I believe all indexes will be recomputed (assuming @ijsnow work on reindexing based on command line flags has gone in). If it hasn't, it will only re-index when there is a new commit to index.

[emidoots]

Note ongoing work here: google/zoekt#86

[emidoots]

@sourcegraph/core-services I am reassigning to you for tracking purposes. I know you're already doing some work on this in google/zoekt#86

It sounds like a good first step here would be what @tsenart suggested in google/zoekt#86 (comment) with just setting GOGC=50 in our zoekt-webserver and zoekt-indexserver containers? Any reason to not do that?

[emidoots]

setting GOGC=50 in our zoekt-webserver and zoekt-indexserver containers

would be good to land this in 3.7 if so

[tsenart]

would be good to land this in 3.7 if so

That sounds like a good idea. Can you land those patches?

[ijt]

I'm afk the rest of the day. If someone else wants to land this for 3.7 I'm fine with that.

…

On Wed, Aug 14, 2019, 2:29 AM Tomás Senart ***@***.***> wrote: would be good to land this in 3.7 if so That sounds like a good idea. Can you land those patches? — You are receiving this because you are on a team that was mentioned. Reply to this email directly, view it on GitHub <https://github.com/sourcegraph/sourcegraph/issues/3792?email_source=notifications&email_token=AAADZKS4Q4TA7MMXZCXKFX3QEPF6JA5CNFSM4HKEZPA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4IHSZQ#issuecomment-521173350>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAADZKWRWXO6NADUSZ6ROB3QEPF6JANCNFSM4HKEZPAQ> .

[emidoots]

(I am doing so)

[ijt]

Sounds good. Thanks!

…

On Wed, Aug 14, 2019, 12:49 PM Stephen Gutekanst ***@***.***> wrote: (I am doing so) — You are receiving this because you are on a team that was mentioned. Reply to this email directly, view it on GitHub <https://github.com/sourcegraph/sourcegraph/issues/3792?email_source=notifications&email_token=AAADZKWD3OTELH35DF3V7YDQEROTZA5CNFSM4HKEZPA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4J5BYQ#issuecomment-521392354>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAADZKSQP6D24GO4NPFL2ELQEROTZANCNFSM4HKEZPAQ> .

[emidoots]

• GOGC=50 set in Docker containers here: https://github.com/sourcegraph/infrastructure/pull/1601
• deploy-sourcegraph image versions bumped here:
  • sourcegraph/deploy-sourcegraph@55fdba2 sourcegraph/deploy-sourcegraph@120b176
  • sourcegraph/deploy-sourcegraph-docker@b638e57
• Companion server/dev change: https://github.com/sourcegraph/sourcegraph/pull/5199

[emidoots]

We'll see how big of an impact this has and, if insufficient, will re-open this.