Queries leveraging search_after fail when fields are not defined by the index document

[jschulist]

🐛 Bug Report

This is outlined and briefly discussed here: elastic/elasticsearch#28806
I encountered the same issue and seeing how the above issue was closed due to "problem is not in es", I'm opening an issue here as IMO it is an issue with how the official elasticsearch js library handles this scenario.

To Reproduce

Query an index that includes documents which do not provide values for all fields.

{
   "size":25,
   "query":{
      "bool":{
         "must":[
            {
               "term":{
                  "someField1":"some-value"
               }
            },
         ]
      }
   },
   "sort":[
      {
         "someSortField1":{
            "order":"asc"
         }
      },
      {
         "someSortField2":{
            "order":"asc"
         }
      },
   ],
}

Using the results of the above query, make a subsequent query providing a search_after sort key based on one of the documents that did not provide values for all fields. For example:

{
   "size":25,
   "query":{
      "bool":{
         "must":[
            {
               "term":{
                  "someField1":"some-value"
               }
            },
         ]
      }
   },
   "sort":[
      {
         "someSortField1":{
            "order":"asc"
         }
      },
      {
         "someSortField2":{
            "order":"asc"
         }
      },
   ],
   "search_after":[
      9223372036854776000,
      "some-name",
   ]
}

Response from the above query indicates a illegal_state_exception "No matching token for number_type [BIG_INTEGER]"

Expected behavior

Ideally the js library handles this scenario without encountering the above issue. If that's not possible, at a minimum I would suggest calling out this limitation in the documentation and providing best practice guidance regarding any known workarounds for how to handle this scenario.

Your Environment

• node version: 8.10
• @elastic/elasticsearch version: ^7.4.0
• os: Linux
• ES cluster version: 7.1

[delvedor]

Hello! This is not a problem strictly related to the client itself but to JavaScript the language.
Every number in JavaScript is a double, and every number inside a JSON will be cast to a double-precision 64-bit binary format IEEE 754.
A possible workaround has been suggested in elastic/kibana#1274 (comment).
For now, the client will not have support for BigInt out of the box, but if you want, you can write a custom parser/serializer to handle them.

[stale]

We understand that this might be important for you, but this issue has been automatically marked as stale because it has not had recent activity either from our end or yours.
It will be closed if no further activity occurs, please write a comment if you would like to keep this going.

Note: in the past months we have built a new client, that has just landed in master. If you want to open an issue or a pr for the legacy client, you should do that in https://github.com/elastic/elasticsearch-js-legacy