[BUG] NPE when queryResults.isEmpty() but hasAggs is true

[janheise]

Describe the bug
We have the following occuring;

If there is an empty result:

OpenSearch/server/src/main/java/org/opensearch/action/search/SearchPhaseController.java

Lines 465 to 467 in 8c9481a

	if (queryResults.isEmpty()) { // early terminate we have nothing to reduce
	final TotalHits totalHits = topDocsStats.getTotalHits();
	return new ReducedQueryPhase(

returns with an almost empty ReducedQueryPhase instance, which further on in

OpenSearch/server/src/main/java/org/opensearch/action/search/QueryPhaseResultConsumer.java

Lines 163 to 165 in 8c9481a

	if (hasAggs) {
	// Update the circuit breaker to replace the estimation with the serialized size of the newly reduced result
	long finalSize = reducePhase.aggregations.getSerializedSize() - breakerSize;

creates an NPE if the query had aggregations.

It is probably related to the multi_terms aggregation.

To Reproduce
Steps to reproduce the behavior:

• Start a new/empty OpenSearch 2.4 instance
• create an index PUT http://localhost:9200/test_jh
• run the following query:

{
	"aggregations": {
		"agg": {
			"date_histogram": {
				"field": "timestamp",
				"format": "date_time",
				"fixed_interval": "30m"
			},
			"aggregations": {
				"agg": {
					"multi_terms": {
						"terms": [
							{
								"field": "field1"
							},
							{
								"field": "field2"
							}
						]
					}
				}
			}
		}
	}
}

The following error occurs:

{
	"error": {
		"root_cause": [
			{
				"type": "response_handler_failure_transport_exception",
				"reason": "java.lang.NullPointerException: Cannot invoke \"org.opensearch.common.io.stream.NamedWriteable.getWriteableName()\" because \"namedWriteable\" is null"
			}
		],
		"type": "search_phase_execution_exception",
		"reason": "",
		"phase": "fetch",
		"grouped": true,
		"failed_shards": [
			{
				"shard": 0,
				"index": "test_jh",
				"node": "NbG0fKofQ5OOkIagnOIXvA",
				"reason": {
					"type": "response_handler_failure_transport_exception",
					"reason": "java.lang.NullPointerException: Cannot invoke \"org.opensearch.common.io.stream.NamedWriteable.getWriteableName()\" because \"namedWriteable\" is null",
					"caused_by": {
						"type": "null_pointer_exception",
						"reason": "Cannot invoke \"org.opensearch.common.io.stream.NamedWriteable.getWriteableName()\" because \"namedWriteable\" is null"
					}
				}
			}
		],
		"caused_by": {
			"type": "null_pointer_exception",
			"reason": "Cannot invoke \"org.opensearch.search.aggregations.InternalAggregations.getSerializedSize()\" because \"reducePhase.aggregations\" is null"
		}
	},
	"status": 500
}

Expected behavior
There should be no NPE, the empty result should be propagated as is.

Host/Environment (please complete the following information):

• OS: macOS, Ubuntu 20.04
• Version 2.4

[janheise]

Added example to reproduce problem.

[dblock]

@janheise You did most of the work to debug this! Thank you. Want to try to implement a unit test and maybe even a fix?

[janheise]

@dblock I am currently too busy to tackle it myself - and am surprised about the good first issue label. From what I've seen (but I did not look into it too deeply), you need a lot of context to decide where the problem actually starts and how a bugfix should look.

[dblock]

@janheise Thanks! @anasalkouz triaged it as such, I imagine because there's a request/response repro (thank you). I think a unit test is not too difficult based on this, but I agree that fixing the problem may be much more involved. Would value a PR that just has the unit test, it will get whomever wants to fix this one step further.

[sarvajeetp]

/assign

[sarvajeetp]

When I tried to debug, it turned out that queryResults has size of 1 and hence it doesn't go into below condition

OpenSearch/server/src/main/java/org/opensearch/action/search/SearchPhaseController.java

Lines 465 to 467 in af566e1

	if (queryResults.isEmpty()) { // early terminate we have nothing to reduce
	final TotalHits totalHits = topDocsStats.getTotalHits();
	return new ReducedQueryPhase(

Instead below aggregation variable evaluates to null, because bufferedAggs list is empty.

OpenSearch/server/src/main/java/org/opensearch/action/search/SearchPhaseController.java

Line 536 in af566e1

final InternalAggregations aggregations = reduceAggs(aggReduceContextBuilder, performFinalReduce, bufferedAggs);

[anasalkouz]

am surprised about the good first issue label. From what I've seen (but I did not look into it too deeply), you need a lot of context to decide where the problem actually starts and how a bugfix should look.

Agreed, will keep the label for the unit test

[geetptl]

Hey @anasalkouz @janheise @dblock, I am going to pick this up since the previous assignee has left. Will reply here if I need any help

[anasalkouz]

Hi @geetptl, it has been a while, are you actively working on this?

[dblock]

There may be a repro in #7089. Help us turn this into a failing unit test?

[z002Holpp]

Could you give a rough estimate about when the issue could be solved or if someone actively works on it? We have some business decisions depending on that.

[dblock]

@z002Holpp I would say nobody is working on it unless it clearly says so, and looks like @geetptl isn't around to confirm they still are :( maybe you can help with some hands?

[geetptl]

Het @dblock, I'm not working on this, feel free to take it up!