High latency when several knn queries in parallel

[joancf]

Hi, I have an index with 2M documents which are distributed in 500K main documents that have 600K nested documents which contain a 1M of nested documents with the embeddings . The ES runs in a docker, with 4Gb of ram assigned to it, The full index is about 50GB and the embeddings are 2.3Gb in disk (1.8Gb the docvectors, according to ES _disk_usage)

I experiment a problem of performance degradation when running queries in parallel. To test it i generate 120 random queries and i launch several in parallel .
So, the basic 120 elastiknn_nearest_neighbors queries (without any extra filter) can be ran in 19 seconds. Adding more parallel processes, the global time decreases to 12Secons (that is 0,1 seconds /query)
If i add a filter to the query then
the 120 queries take 26 seconds when run one after the other
it reduces to 15 seconds when running 2 in parallel,
but increases to 64 seconds when running 4 or more in parallel, This number of 6x seconds is stable on different experiments and changing the number of queries in parallel). But sometimes the global time of running the 120 queries with a high number of parallelism goes back to 10Seconds.
Analyzing the time of each query in, most cases there are 1 or 2 queries that take 60 seconds while the others still run fast.
checking the system paramenters

• the cpu, it's clear that when this happens the cpu load is low,
• kibana does not show any extra activity, not a GC or disk iom

So, I would like to ask if you have any idea of which can be the issue and if there is a way to solve it?

Thanks

[alexklibisz]

Hi, I have an index with 500K documents with 600K nested documents which contain a 1M of nested documents with embeddings

I don't understand this part. I don't understand how there can be 500k docs, and 600k docs, and 1M docs? Maybe you can restate it?

In general I recommend attaching a profiler like visual VM. The docker-compose.benchmarking.yaml shows an example with the JAVA_OPTS necessary to expose JMX: https://github.com/alexklibisz/elastiknn/blob/main/docker/docker-compose.benchmarking.yaml#L18

Then you would attach VisualVM to localhost:6000. I generally use the sampling functionality to look for any obvious bottlenecks.