Searching in a json field path1.path2.path3:value should return a result

[fmassot]

Currently this does not work as tantivy is doing something slightly different when writing and when searching.

Let's take a document to index that is very common in the OpenTelemetry world:

{"resource": {"k8s.container.name": "prometheus"}}

The doc_mapping is as follows:

doc_mapping:
   - name: resource
      type: json

When writing the index, tantivy will interpret k8s.container.name as the field name and will store that as is along with the value (and the type).
When searching document matching the query resource.k8s.container.name:prometheus, quickwit will remove the resource part and give tantivy this term to match k8s.container.name:prometheus. The issue is that tantivy will interpret the dots as a separator and will build a term with an internal separator and this won't match what was written previously.

I suggest modifying a bit how tantivy writes the JSON terms by using the dots in field names to define segment path.

This approach is not perfect as this will allow mixing {"resource": {"k8s.container.name": "prometheus"}} and {"resource": {"k8s": {"container": {"name": "prometheus"}}}} but having dots in fieldnames is very common in the log world and it will be very painful to escape dots.

@fulmicoton what do you think?

[fulmicoton]

resource.k8s\.container\.name:prometheus should work.

I agree it is tempting to have resource.k8s.container.name:prometheus but it has some downside (ambiguity/shadowing).

[fmassot]

I tried to escape the dot but... it does not seem to work. When looking at tantivy code, I saw this json_path.split('.') so it will still split on the dot. Or I may be missing something.

https://github.com/quickwit-oss/tantivy/blob/9a090ed994708c220e574f1bf9ad05ce419c10a8/src/indexer/json_term_writer.rs#L272

[fulmicoton]

we can call it a bug then :)

[fulmicoton]

Fix in quickwit-oss/tantivy#1682