Painless script_fields don't have access to a _source variable

[Bargs]

Elasticsearch version: master

Plugins installed: none

JVM version: 1.8.0_60

OS version: OSX El Capitan

Description of the problem including expected versus actual behavior:

According to the docs script_fields can access the document _source via a _source keyword. When attempting this with a Painless script, I get a Variable [_source] is not defined. error.

Steps to reproduce:

1. Execute a query with a painless script_field that attempts to access _source, something like this:

    "script_fields": {
        "scriptip": {
            "script": {
                "inline": "return _source.ip;",
                "lang": "painless"
            }
        }
    }

Provide logs (if relevant):

{
  "error": {
    "root_cause": [
      {
        "type": "script_exception",
        "reason": "compile error",
        "script_stack": [
          "return _source.ip;",
          "       ^---- HERE"
        ],
        "script": "return _source.ip;",
        "lang": "painless"
      }
    ],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "phase": "query_fetch",
    "grouped": true,
    "failed_shards": [
      {
        "shard": 0,
        "index": "logstash-0",
        "node": "00s71MPnSRenvz6mflLQeQ",
        "reason": {
          "type": "script_exception",
          "reason": "compile error",
          "caused_by": {
            "type": "illegal_argument_exception",
            "reason": "Variable [_source] is not defined."
          },
          "script_stack": [
            "return _source.ip;",
            "       ^---- HERE"
          ],
          "script": "return _source.ip;",
          "lang": "painless"
        }
      }
    ],
    "caused_by": {
      "type": "script_exception",
      "reason": "compile error",
      "caused_by": {
        "type": "illegal_argument_exception",
        "reason": "Variable [_source] is not defined."
      },
      "script_stack": [
        "return _source.ip;",
        "       ^---- HERE"
      ],
      "script": "return _source.ip;",
      "lang": "painless"
    }
  },
  "status": 500
}

Describe the feature:

[jdconrad]

_source is definitely a controversial issue, but as of right now, _source is only supported in update scripts within Painless.

[Bargs]

Sounds like the script fields docs should be updated to at least note that then. Right now they're a bit misleading because the doc example is in Painless and the _source example doesn't mention the script lang explicitly.

[jdconrad]

Yeah, I agree completely that the documentation needs to be improved. I will be investing some significant effort into this (at least for the Painless side) over the next few weeks.

[rmuir]

I dont think it needs to be controversial for script fields? I think the fix is to move them to the ExecutableScript api, so they have _source and other stuff.

The problem is, they are using the SearchScript api today, which should be for the millions-of-documents case

[clintongormley]

I think the fix is to move them to the ExecutableScript api, so they have _source and other stuff.

+1 It'll be a breaking change for existing scripts which currently access _source. and will need to change to ctx._source., but I think it's the right move.

[rmuir]

Right, "breaking change" is one way to look at it. "broken now" is another :)

[clintongormley]

@Bargs @epixa Could you give us some examples of how people use script fields? Do people use them to extract values from the source, or from doc values?

[Bargs]

@clintongormley definitely doc values up until now. Since Lucene expressions only support numerical data I don't think there was ever much point in accessing source. In fact, it looks like Kibana throws an error if you try to access _source in an expression script. With painless there might be more reason to use source, but it's hard to say how users will leverage it.

An additional (possible) complication is that Kibana scripted fields are made available on the Visualize tab as well, so the scripts can be used in aggregations. Is it possible to access source in scripts on aggregations?

[nik9000]

Is it possible to access source in scripts on aggregations?

In scripts it is possible. Groovy supports it now and painless doesn't. It is always much, much, much slower than using doc_values. It is a speed vs flexibility tradeoff vs education thing.