Debug logs for performance assessment

[ezorita]

I am interested in getting some insights about the performance in query time. Is there a way to configure the library to display logs that can give insights on performance? For a given query, metrics that would be helpful are:

• How many LSH candidates are matched.
• How many vectors will be pulled from _source to compute distances. Does store: true improve performance of this step?
• How long does it take to compute the cosine distance.
• Memory usage in all/any of these steps.

Is any of this even possible?

Thank you in advance!

[alexklibisz]

Hi, this kind of logging is not currently implemented. It wouldn't be particularly difficult to add timing logs. Basically just find the spots in the code worth timing and make sure to use the loggers provided by elasticsearch and lucene.

I don't know of a clean way to track memory usage for just Elastiknn. You can of course use metrics tools already built into the JVM. That would give you memory usage of the entire Elasticsearch server.

[alexklibisz]

@ezorita if you're interested in trying to add this, there's actually some commented-out code you can start with as an example:

elastiknn/elastiknn-lucene/src/main/java/org/apache/lucene/search/MatchHashesAndScoreQuery.java

Lines 97 to 101 in 9e320dd

	// if (counter.numHits() < candidates) {
	// logger.warn(String.format(
	// "Found fewer approximate matches [%d] than the requested number of candidates [%d]",
	// counter.numHits(), candidates));
	// }

There's a developer guide here: https://github.com/alexklibisz/elastiknn/tree/main/development