Allow field mappers to retrieve fields from source.

[jtibshirani]

This PR adds new method FieldMapper#lookupValues(SourceLookup) that extracts and parses the source values. This lets us return values like numbers and dates in a consistent format, and also handle special data types like constant_keyword. The lookupValues method calls into parseSourceValue, which mappers can override to specify how values should be parsed.

Older draft PR: #56473

Relates to #55363.

[elasticmachine]

Pinging @elastic/es-search (:Search/Mapping)

[jtibshirani]

As part of this PR, we need to decide on the 'standard format' for each field type. I'll work to collect feedback/ discussion on this point. Here's the current proposal:

• boolean: parse into boolean
• binary: no change to value
• completion: no change to value
• constant_keyword: don't consult source, return the constant value
• date: parse to date string based on mapping format
• date_nanos: parse to long nanos based on mapping format
• ip: collapse 0s in IPv6 addresses (to be consistent with docvalues_fields)
• keyword: convert value to string, do not apply normalization
• numeric types (integer, float, etc.): parse value to the right type, for example for a long field "42.3" becomes 42
• range types (integer_range, etc.): parse each bound to the right type, this includes both numbers, dates, and IPs
• rank_feature: convert value to float
• scaled_float: convert value to float, taking scaling factor into account
• text, annotated_text: convert value to string

Some notable examples where we don't parse at all:

• histogram, rank_features, vector: no change to value, don't attempt to convert each component into a number. For me the complexity of digging through and standardizing these structured values outweighed the benefit.
• murmur3, token_count: simply convert to string. It didn't seem practical to always recompute a hash/ token count, and I couldn't see an important use case for returning these values.

We never return a field that has been dropped through ignore_malformed. This way, the values returned for particular field will always have the same type and format. The idea is that the value is ignored in all high-level parts of the search request, including the query, aggs, and field retrieval.

Geo fields aren't handled yet, I plan to do this in a follow-up PR where we can have a dedicated discussion.

[nik9000]

date: parse to long millis based on mapping format

This is one place where format is pretty important, I think. We should probably apply whatever the default "format" is for this field. Though that can totally be a follow up.

Also in a follow up, folks should be able to specify the format, I think. That way folks can say things like "give this is millis-since-epoch even if the default format is ISO8601".

I'm not sure if we need time zone support too. I know the aggs have it, but that feels like a big much at this point.

[nik9000]

ip: no change to value

Should this canonicalize the IP?

[jtibshirani]

This is one place where format is pretty important, I think. We should probably apply whatever the default "format" is for this field. Though that can totally be a follow up.

I'll handle this now, I do think it fits best within this PR!

Also in a follow up, folks should be able to specify the format, I think. That way folks can say things like "give this is millis-since-epoch even if the default format is ISO8601".

+1, this is tracked in the meta-issue as a follow-up (#55363).

Should this canonicalize the IP?

Sorry if I'm missing something obvious, what's an example of the canonicalization you have in mind ?

[nik9000]

Sorry if I'm missing something obvious, what's an example of the canonicalization you have in mind ?

I believe round tripping IPs through doc values would apply this.

[nik9000]

I believe round tripping IPs through doc values would apply this.

I think it might be as simple as parsing and toStringing to make that happen though.

[jtibshirani]

A heads up that I rebased and force-pushed to incorporate a refactor (#56976).

[astefan]

LGTM.
Left few minor comments.

[jtibshirani]

Thanks @astefan for the review!

@nik9000 it would be great to get your review as well. Here are the changes since last time:

• We now format dates using the format in the mapping. This applies to date, date_nanos, and date_range.
• IPs are standardized using the utilities in InetAddresses. I don't have a strong intuition here, but it felt like a good choice since it matches what's returned in docvalue_fields.

[nik9000]

LGTM! Sorry for the delay.