Slowness when doing basic queries

[jeremyaouad]

I was testing with Influx 2.3 container.
I have a set of 80 metrics, emitted every 10 seconds, for about 500 processes => cardinality of 500*80 = 40000.
I didn't put any limits on the influxdb container cpu and memory.
The machine I'm testing on has 28 cores and 500GB ram. I'm using the ram as storage for influxdb (bbolt mounted on a tmpfs volume), with a read/write speed of 3 GB/sec.
I wrote 12 very basic queries of the sort "select * from x where y = z" in a file and launched them periodically, every 10 sec, and timed the duration of the execution. In the query, I set the range for the past 10 seconds. The returned data set would be between 200 and 400 records.

I observed the following:

• For the write performance, the cpu went up to 5-10% max. Memory was also good.
• For the read performance, whenever the script was launched, the cpu usage went up to 12 cores, memory was ok. However, the execution time kept on increasing from a couple of seconds to 2 min as time passed. During the execution time, the cpu usage would be 12 cores.
• What puzzled me more is the following: I stopped emitting metrics in influxdb, I queried for the past 10 seconds, and I was still getting a 2 min query time.

I also launched the queries with the profiler output. The execution time was taking all of the time. Specifically this:
influxdb.readWindowAggregateSource,ReadWindowAggregateByTime11

Is there something I'm missing?

Thanks.