NPE in SourceConfirmedTextQuery

[ebadyano]

Elasticsearch Version

main

Installed Plugins

No response

Java Version

bundled

OS Version

irrelevant

Problem Description

NPE in SourceConfirmedTextQuery

org.elasticsearch.search.query.QueryPhaseExecutionException: Query Failed [Failed to execute main query]
	at org.elasticsearch.server@9.1.0/org.elasticsearch.search.query.QueryPhase.addCollectorsAndSearch(QueryPhase.java:237)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.search.query.QueryPhase.executeQuery(QueryPhase.java:143)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:70)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$39(IndicesService.java:1669)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$40(IndicesService.java:1735)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:165)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:148)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:440)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:111)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1741)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1663)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:670)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:877)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.search.SearchService.lambda$executeQueryPhase$7(SearchService.java:710)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:79)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:76)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:101)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)
	at java.base/java.lang.Thread.run(Thread.java:1447)
Caused by: java.lang.NullPointerException: Cannot invoke "java.util.List.iterator()" because "values" is null
	at org.elasticsearch.mapper.extras@9.1.0/org.elasticsearch.index.mapper.extras.SourceConfirmedTextQuery$RuntimePhraseScorer.getOrCreateMemoryIndex(SourceConfirmedTextQuery.java:437)
	at org.elasticsearch.mapper.extras@9.1.0/org.elasticsearch.index.mapper.extras.SourceConfirmedTextQuery$RuntimePhraseScorer.computeFreq(SourceConfirmedTextQuery.java:448)
	at org.elasticsearch.mapper.extras@9.1.0/org.elasticsearch.index.mapper.extras.SourceConfirmedTextQuery$RuntimePhraseScorer.freq(SourceConfirmedTextQuery.java:426)
	at org.elasticsearch.mapper.extras@9.1.0/org.elasticsearch.index.mapper.extras.SourceConfirmedTextQuery$RuntimePhraseScorer$1.matches(SourceConfirmedTextQuery.java:386)
	at org.apache.lucene.core@10.2.1/org.apache.lucene.search.DenseConjunctionBulkScorer.scoreWindowUsingLeapFrog(DenseConjunctionBulkScorer.java:353)
	at org.apache.lucene.core@10.2.1/org.apache.lucene.search.DenseConjunctionBulkScorer.scoreWindow(DenseConjunctionBulkScorer.java:214)
	at org.apache.lucene.core@10.2.1/org.apache.lucene.search.DenseConjunctionBulkScorer.score(DenseConjunctionBulkScorer.java:132)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.search.internal.CancellableBulkScorer.score(CancellableBulkScorer.java:46)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:460)
	at org.apache.lucene.core@10.2.1/org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:809)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:387)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$3(ContextIndexSearcher.java:365)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:328)
	at org.apache.lucene.core@10.2.1/org.apache.lucene.search.TaskExecutor$Task.run(TaskExecutor.java:173)
	at org.apache.lucene.core@10.2.1/org.apache.lucene.search.TaskExecutor.invokeAll(TaskExecutor.java:111)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:369)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:336)
	at org.elasticsearch.server@9.1.0/org.elasticsearch.search.query.QueryPhase.addCollectorsAndSearch(QueryPhase.java:212)
	... 23 more

Steps to Reproduce

Found in serverless log alerts

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/es-search-relevance (Team:Search Relevance)

[pmpailis]

Handling the NPE seems rather straightforward, but I've been unable to reproduce the above error.
Looking at the stacktrace, we have:

Caused by: java.lang.NullPointerException: Cannot invoke "java.util.List.iterator()" because "values" is null
	at org.elasticsearch.mapper.extras@9.1.0/org.elasticsearch.index.mapper.extras.SourceConfirmedTextQuery$RuntimePhraseScorer.getOrCreateMemoryIndex(SourceConfirmedTextQuery.java:437)
	at org.elasticsearch.mapper.extras@9.1.0/org.elasticsearch.index.mapper.extras.SourceConfirmedTextQuery$RuntimePhraseScorer.computeFreq(SourceConfirmedTextQuery.java:448)
	at org.elasticsearch.mapper.extras@9.1.0/org.elasticsearch.index.mapper.extras.SourceConfirmedTextQuery$RuntimePhraseScorer.freq(SourceConfirmedTextQuery.java:426)
	at org.elasticsearch.mapper.extras@9.1.0/org.elasticsearch.index.mapper.extras.SourceConfirmedTextQuery$RuntimePhraseScorer$1.matches(SourceConfirmedTextQuery.java:386)

however, looking at how the null values are generated, they come from a predefined valueFetcher.

 List<Object> values = valueFetcher.apply(docID());

For SourceConfirmedTextQuery, AFAICT the codepath is exercised only for highlighting, and this valueFetcher is explicitly set as SourceValueFetcher for non synthetic source indices, which in turn can not return null.

For the specific exception above however, the index in question did use synthetic source as per the index.mapping.source.mode

          "source": {
            "mode": "synthetic"
          }

This in turn implies that this code path is exercised for standard matches as well. Looking into the different implementations of org.elasticsearch.index.mapper.ValueFetcher#fetchValues , the 2 cases where this could return null are StoredValueFetcher and OffsetSourceFieldType. The latter is not exposed and is used internally by semantic text fields, which is not the case here.

So looking into the StoredValueFetcher, this could return null causing the above NPE under the following conditions:

        List<Object> values = leafSearchLookup.fields().get(fieldname).getValues();
        if (values == null) {
            return values;
...

However, I haven't been able to exercise the above codepath yet. Will take another look tomorrow, before setting up a PR to address the actual NPE.

[pmpailis]

Was finally able to reproduce the NPE above, albeit from a different codepath, in MatchOnlyTextFieldMapper.MatchOnlyTextFieldType#valueFetcher

            if (searchExecutionContext.isSourceSynthetic()) {
                String name = storedFieldNameForSyntheticSource();
                StoredFieldLoader loader = StoredFieldLoader.create(false, Set.of(name));
                return context -> {
                    LeafStoredFieldLoader leafLoader = loader.getLoader(context, null);
                    return docId -> {
                        leafLoader.advanceTo(docId);
                        return leafLoader.storedFields().get(name);
                    };
                };
            }

The following trigger a null value for the above, which would in turn throw an NPE during match

DELETE my-index

PUT my-index
{
    "settings": {
        "index.mapping.source.mode": "synthetic"
    },
    "mappings": {
        "properties": {
            "user": {
              "properties": {
                "name": {
                  "type": "keyword",
                  "fields": {
                    "text": {
                      "type": "match_only_text"
                    }
                  }
                }
              }
            }
        }
    }
}

POST my-index/_doc/
{
    "user": {
        "name": "you know, for search"
    }
}

GET my-index/_search
{
    "query": {
        "match_phrase": {
            "user.name.text": "you know"
        }
    }
}

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)

[martijnvg]

The cause seems to be #129126, if multi fields are of type match_only field type, then we no longer store the field. Given that the parent field stored it already. The SourceConfirmedTextQuery should source the field form parent field instead in that case.