nested/object field access in scripts should throw an appropriate error

[rjernst]

When trying to access a nested or object field within a script, the behavior is different depending on the method called. When trying to retrieve the value of the field, an exception is thrown saying the field does not exist (when it fact it is simply not a field mapper). If you try to check if the doc contains the field, it simply returns false, again incorrectly since the field does exist, just not as a field mapper. I think we should throw an error in both these cases, but with a good message that indicates the field exists but is not accessible in scripts. Note that I'm not sure what happens when trying to access a concrete field that is parented by a nested field. I believe I have seen issues for it before where the field just comes back as no values. I think we should throw an error in this case as well.

[clintongormley]

+1 makes sense to give a useful exception when trying to access nested/object fields.

If you try to access a concrete field under nested today, it might succeed if include_in_parent/root are enabled. I think once include_in_parent/root are removed (see #12461) then we could throw an exception for concrete nested fields too.

[glefloch]

Hi, can I take this issue?

[MrOrz]

I wonder if this statement is still valid for Elasticsearch 6.1.0:

When trying to access a nested or object field within a script, the behavior is different depending on the method called. When trying to retrieve the value of the field, an exception is thrown saying the field does not exist (when it fact it is simply not a field mapper). If you try to check if the doc contains the field, it simply returns false, again incorrectly since the field does exist, just not as a field mapper.

In Elasticsearch 6.1.0, actually we can read and write nested objects in scripts, with no include_in_root nor include_in_parent in place.

Proof of concept

1. Start a elasticsearch server on port 9200
2. Follow the nested object guide to setup indexes and index an document:

curl -XPUT 'localhost:9200/my_index?pretty' -H 'Content-Type: application/json' -d'
{
  "mappings": {
    "my_type": {
      "properties": {
        "user": {
          "type": "nested" 
        }
      }
    }
  }
}
'
curl -XPUT 'localhost:9200/my_index/my_type/1?pretty' -H 'Content-Type: application/json' -d'
{
  "group" : "fans",
  "user" : [
    {
      "first" : "John",
      "last" :  "Smith"
    },
    {
      "first" : "Alice",
      "last" :  "White"
    }
  ]
}
'

3. Mutate the indexed document's nested field with scripted update:

curl -POST 'localhost:9200/my_index/my_type/1/_update?pretty' -H 'Content-Type: application/json' -d'
{
  "script": {
    "source": "ctx._source.user.get(1).put(\"last\", \"SmithEX\");",
    "lang": "painless"
  }
}
'

4. See the result:

curl -XGET 'localhost:9200/my_index/my_type/1?pretty'

It returns:

{
  "_index" : "my_index",
  "_type" : "my_type",
  "_id" : "1",
  "_version" : 2,
  "found" : true,
  "_source" : {
    "group" : "fans",
    "user" : [
      {
        "first" : "John",
        "last" : "Smith"
      },
      {
        "first" : "Alice",
        "last" : "SmithEX"
      }
    ]
  }
}

Notice that the second user's last name is correctly mutated.

[rjernst]

@MrOrz nested documents are always available via _source. This issue is about accessing them from docvalues within a scoring script for search.

[MrOrz]

I see. Didn't notice the difference in subjects.
Thanks for the explanation!

[stu-elastic]

In LeafDocLookup we should not return an assessor for nested or Object fields. Then when executing the script we can raise the appropriate error. It's possible to do this all within the get method of LeafDocLookup.

[Shitai-Roblox]

Hi Folks,

I am thinking if we support this now? I came from this https://discuss.elastic.co/t/can-not-access-array-with-objects-in-painless/198226.

[thecoop]

@rjernst Is this still something we're going to fix at some point?

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)