Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

QL: IndexResolver issue when dealing with dotted fields #100869

Closed
costin opened this issue Oct 14, 2023 · 3 comments · Fixed by #100875
Closed

QL: IndexResolver issue when dealing with dotted fields #100869

costin opened this issue Oct 14, 2023 · 3 comments · Fixed by #100875
Assignees
Labels
:Analytics/ES|QL AKA ESQL blocker >bug Team:QL (Deprecated) Meta label for query languages team

Comments

@costin
Copy link
Member

costin commented Oct 14, 2023

Description

IndexResolver tries to create a tree structure for field names that contain dots (e.g. ECS) which can conflict with fields in other indices in which case all underlying fields end up being discarded as unsupported.

Consider field "file.name" as keyword in "index-1" and "file" as keyword in "index-2". This results in the following mapping from field caps:

"indices": ["index-1", "index-2"],
"fields":
{
 "file.name":
 {
  "keyword":
  {
   "type": "keyword",
   "metadata_field": false,
   "searchable": true,
   "aggregatable": true,
   "indices": ["index-1"]
  },
  "unmapped":
  {
   "type": "unmapped",
   "metadata_field": false,
   "searchable": false,
   "aggregatable": false,
   "indices": ["index-2"]
  }
 },
 "file":
 {
  "keyword":
  {
   "type": "keyword",
   "metadata_field": false,
   "searchable": true,
   "aggregatable": true,
   "indices": ["index-2"]
  },
  "object":
  {
   "type": "object",
   "metadata_field": false,
   "searchable": false,
   "aggregatable": false,
   "indices": ["index-1"]
  }
 }
}

Notice that file is mapped both as an object in index-1 and keyword in index-2. IndexResolver sees the conflict and thus discards as unsupported all the fields under it, including file.name hence why the ESQL query where file.name is not null fails.
When a projection is added where file.name is not null | keep file.name, IndexResolver doesn't ask the mapping information for file which removes the conflict and thus file.name can be resolved.

The main fix here is to NOT discard fields under an object leaf as unsupported if they have no conflict. In this example file.name is mapped as keyword in one index and unmapped in the other so across all indices it will have the same type.
Furthermore listing the field as missing instead of say unsupported is incorrect.

@costin costin added >bug blocker Team:QL (Deprecated) Meta label for query languages team :Analytics/ES|QL AKA ESQL labels Oct 14, 2023
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-ql (Team:QL)

@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/elasticsearch-esql (:Query Languages/ES|QL)

costin added a commit to costin/elasticsearch that referenced this issue Oct 15, 2023
In certain scenarios, a field can be mapped both as a primitive and
 object, causing it to be marked as unsupported, losing any potential
 subfields that might have been discovered before.
This commit preserve them to avoid subfields from being incorrectly
 reported as missing.

Fix elastic#100869
@astefan
Copy link
Contributor

astefan commented Oct 16, 2023

listing the field as missing instead of say unsupported is incorrect

This should only be valid for ESQL. SQL hides these fields.
There's also the use case of show columns where these fields should be displayed.

costin added a commit that referenced this issue Oct 16, 2023
In certain scenarios, a field can be mapped both as a primitive and
 object, causing it to be marked as unsupported, losing any potential
 subfields that might have been discovered before.
This commit preserve them to avoid subfields from being incorrectly
 reported as missing.

Fix #100869
costin added a commit to costin/elasticsearch that referenced this issue Oct 16, 2023
In certain scenarios, a field can be mapped both as a primitive and
 object, causing it to be marked as unsupported, losing any potential
 subfields that might have been discovered before.
This commit preserve them to avoid subfields from being incorrectly
 reported as missing.

Fix elastic#100869
elasticsearchmachine pushed a commit that referenced this issue Oct 16, 2023
In certain scenarios, a field can be mapped both as a primitive and
 object, causing it to be marked as unsupported, losing any potential
 subfields that might have been discovered before.
This commit preserve them to avoid subfields from being incorrectly
 reported as missing.

Fix #100869
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Analytics/ES|QL AKA ESQL blocker >bug Team:QL (Deprecated) Meta label for query languages team
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants