Get an warning: ResponseError: too_long_frame_exception

[barwalker]

Hi all, I encountered with a "too_long_frame_exception" warning when using scroll to fetch data from Elastic.

Issue

Here is the query to get data in 1 month:

                "gte": "2019-11-17T17:00:00.000Z",
                "lte": "2019-12-17T17:00:00.000Z"

GET band-monitor-*/_search
{
    "query": {
      "bool": {
        "must": [
          {
            "bool": {
              "should": [
                {
                  "match": {
                    "band_id": 118217
                  }
                },
                {
                  "match": {
                    "band_id": 118220
                  }
                },
                {
                  "match": {
                    "band_id": 118221
                  }
                }
              ]
            }
          },
          {
            "range": {
              "createdDate": {
                "gte": "2019-11-17T17:00:00.000Z",
                "lte": "2019-12-17T17:00:00.000Z"
              }
            }
          }
        ],
        "filter": {
          "script": {
            "script": {
              "source": "return doc['scan_id'].value % 10 == 0"
            }
          }
        }
      }
    },
    "sort": [
      {
        "scan_id": {
          "order": "asc"
        }
      }
    ],
    "aggs": {},
    "size": 9999
}

Run in Kibana and It returns 13384 records. I think this amount is not big at all.
However when using npm @elastic/elasticsearch in nodejs, it shows an error:

{ ResponseError: too_long_frame_exception
    at IncomingMessage.response.on (/home/admin1/Documents/backend_v1/node_modules/@elastic/elasticsearch/lib/Transport.js:287:25)
    at IncomingMessage.emit (events.js:203:15)
    at IncomingMessage.EventEmitter.emit (domain.js:448:20)
    at endReadableNT (_stream_readable.js:1143:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)
  name: 'ResponseError',
  meta:
   { body: { error: [Object], status: 400 },
     statusCode: 400,
     headers:
      { 'content-type': 'application/json; charset=UTF-8',
        'content-length': '211' },
     warnings: null,
     meta:
      { context: null,
        request: [Object],
        name: 'elasticsearch-js',
        connection: [Object],
        attempts: 0,
        aborted: false } } }

With a smaller range, it works fine.
E.g:
"gte": "2019-11-17T17:00:00.000Z",
"lte": "2019-11-27T17:00:00.000Z"

Expected behaviour

Just 20k data, this package should be working probably. I even try to use scroll with scroll time: 30s, size: 100.
Please help to find any solution for this issue. Thank all.

Your Environment

Node version: 8.10
@elastic/elasticsearch version: ^7.5.0
OS: Ubuntu 16.04
ES cluster version: 6.8

[delvedor]

Hello!
Can you paste here the javascript code that you are using to run the query?
ResponseError means that the error is happening on the Elaticsearch side.

Furthermore, you should use the same major version of Elasticsearch in the client, you can do that by running:

npm i @elastic/elasticsearch@6

[stale]

We understand that this might be important for you, but this issue has been automatically marked as stale because it has not had recent activity either from our end or yours.
It will be closed if no further activity occurs, please write a comment if you would like to keep this going.

Note: in the past months we have built a new client, that has just landed in master. If you want to open an issue or a pr for the legacy client, you should do that in https://github.com/elastic/elasticsearch-js-legacy

[nelsou]

Hello @BernardIsibor
Did you fix your issue ? I'm having exactly the same problem.

too_long_frame_exception when scrolling (by 2.000) over my results (~1.000.000)

@elastic/elasticsearch version: ^6
ES cluster version: 6.5

Please let me know

[jdelriops]

In my case, that error happened because the API was trying to set the scroll_id in the queryString as part of the URL, and that has limit.

I solved the problem changing from:

response = await client.scroll({
  scrollId: my_scroll_id,
  scroll: scroll_params
})

To:

response = await client.scroll({
  method: 'POST',
  body: {
    scroll_id: my_scroll_id,
    scroll: scroll_params
  }
})

That way you can prevent the API code to try to set the scroll_id in the queryString, using the POST body instead.

[lijianl]

In my case, that error happened because the API was trying to set the scroll_id in the queryString as part of the URL, and that has limit.

I solved the problem changing from:

response = await client.scroll({
  scrollId: my_scroll_id,
  scroll: scroll_params
})

To:

response = await client.scroll({
  method: 'POST',
  body: {
    scroll_id: my_scroll_id,
    scroll: scroll_params
  }
})

That way you can prevent the API code to try to set the scroll_id in the queryString, using the POST body instead.

In my case, that error happened because the API was trying to set the scroll_id in the queryString as part of the URL, and that has limit.

I solved the problem changing from:

response = await client.scroll({
  scrollId: my_scroll_id,
  scroll: scroll_params
})

To:

response = await client.scroll({
  method: 'POST',
  body: {
    scroll_id: my_scroll_id,
    scroll: scroll_params
  }
})

That way you can prevent the API code to try to set the scroll_id in the queryString, using the POST body instead.

very impressive