Add ExistsQuery for finding all documents where a certain field exists

[Hodkinson]

Sometimes it's necessary to find all documents that simply contain a field, regardless of its value

An ExistsQuery could be added that allows querying for documents containing a certain field.

Currently RegexQuery::from_pattern(".*", field)? can be used for this, and probably a RangeQuery for numeric fields.

[shikhar]

Another approach could be to model with a separate field that has a keyword indicating that this data is present. This should perform better since don't need to iterate over the term dictionary and all the postings to accumulate a bitset eagerly at query time. You can then simply use a TermQuery that will access the postings list for this keyword.

[PSeitz]

We should have an ExistsQuery similar to AllQuery, that can be optionally negated for a not exists.

There are two ways to get this:

1. Similar to FuzzyTermQuery or RegexQuery. Build a BitSetDocSet upfront by scanning all terms and get their docs from the posting list.
2. Use the upcoming fast field index with support for null values. The fast field index can tell which docs have values. In contrast it can be lazy iterated, no need to construct a BitSetDocSet.

[fulmicoton]

This has been demanded by several quickwit users already.
@shikhar solution is the way to go here:
Add an inverted list for the existence of the field. The place where to store this field is TBD.
It could be a special _existence field , where the values are the (field_name, field_type) for instance.
The posting list format is a tiny bit lame, as it still takes 1 bit per doc/field if when the field is saturated.

(@PSeitz not sure why you want a fast field for this...)

[PSeitz]

There are different ways/places where to store that information and fast field index already has that info. With the typing in columnar, you could also cover exists_ip, exists_string, exists_number, which will be more important when combining multiple data source into a single index via JSON fields.

[williamho]

this exists now, as of #2160:

• Initial Implementation of ExistsQuery #2160

though it doesn't seem like it supports the "optionally negated for a not exists" functionality mentioned above (in #1833 (comment)), which would be nice to have

[fulmicoton]

@williamh thank you for spotting this! Closing the issue.