[Reporting] Using migrated saved search for CSV Export throws errors

[tsullivan]

Kibana version: 8.0/master

Elasticsearch version: 8.0/master

Server OS version: Darwin

Browser version: Chrome

Browser OS version: Darwin

Original install method (e.g. download page, yum, from source, etc.): source / dev mode

Describe the bug:
Appears that the data model for saved search objects changed in 7.4.0. Exporting CSV from a saved search ID had to be fixed in #42964. But if we use a saved search that came from an upgrade, or has the older data model, a similar still incompatibility exists as the original bug.

Steps to reproduce:

1. Create a saved search and dashboard that has a saved search panel in Kibana 7.3.0
2. Upgrade the stack to 7.4.0 and retain Kibana saved objects
3. Follow the instructions in [Feature/Reporting] Export Saved Search CSV as Dashboard Panel Action #34571 and add
   xpack.reporting.csv.enablePanelActionDownload: true in the kibana.yml.
4. Open the dashboard
5. Click Download CSV in the saved search panel

Expected behavior:
Download the CSV

Screenshots (if relevant):

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

server    log   [14:47:55.045] [error][reporting][savedobject-csv] Generate CSV Error! [x_content_parse_exception] [1:580] [field_sort] failed to parse field [order] :: {"path":"/babynames/_search","query":{"scroll":"30s","size":500},"body":"{\"_source\":{\"includes\":[\"date\",\"_source\"]},\"docvalue_fields\":[\"date\"],\"query\":{\"bool\":{\"must\":[],\"filter\":[{\"match_all\":{}},{\"range\":{\"date\":{\"format\":\"strict_date_time\",\"gte\":\"1974-01-01T00:00:00-07:00\",\"lte\":\"2019-08-13T14:47:54-07:00\"}}},{\"bool\":{\"must\":[],\"filter\":[{\"match_all\":{}},{\"match_all\":{}},{\"range\":{\"date\":{\"format\":\"strict_date_optional_time\",\"gte\":\"1974-01-01T07:00:00.000Z\",\"lte\":\"2019-08-13T21:47:54.926Z\"}}}],\"should\":[],\"must_not\":[]}}],\"should\":[],\"must_not\":[]}},\"script_fields\":{},\"sort\":[{\"date\":{\"order\":\"desc\",\"unmapped_type\":\"boolean\"}},{\"d\":{\"order\":\"a\"}},{\"d\":{\"order\":\"e\"}}]}","statusCode":400,"response":"{\"error\":{\"root_cause\":[{\"type\":\"x_content_parse_exception\",\"reason\":\"[1:580] [field_sort] failed to parse field [order]\"}],\"type\":\"x_content_parse_exception\",\"reason\":\"[1:580] [field_sort] failed to parse field [order]\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"No enum constant org.elasticsearch.search.sort.SortOrder.A\"}},\"status\":400}"}
server   error  [14:47:54.940]  [x_content_parse_exception] [1:580] [field_sort] failed to parse field [order] :: {"path":"/babynames/_search","query":{"scroll":"30s","size":500},"body":"{\"_source\":{\"includes\":[\"date\",\"_source\"]},\"docvalue_fields\":[\"date\"],\"query\":{\"bool\":{\"must\":[],\"filter\":[{\"match_all\":{}},{\"range\":{\"date\":{\"format\":\"strict_date_time\",\"gte\":\"1974-01-01T00:00:00-07:00\",\"lte\":\"2019-08-13T14:47:54-07:00\"}}},{\"bool\":{\"must\":[],\"filter\":[{\"match_all\":{}},{\"match_all\":{}},{\"range\":{\"date\":{\"format\":\"strict_date_optional_time\",\"gte\":\"1974-01-01T07:00:00.000Z\",\"lte\":\"2019-08-13T21:47:54.926Z\"}}}],\"should\":[],\"must_not\":[]}}],\"should\":[],\"must_not\":[]}},\"script_fields\":{},\"sort\":[{\"date\":{\"order\":\"desc\",\"unmapped_type\":\"boolean\"}},{\"d\":{\"order\":\"a\"}},{\"d\":{\"order\":\"e\"}}]}","statusCode":400,"response":"{\"error\":{\"root_cause\":[{\"type\":\"x_content_parse_exception\",\"reason\":\"[1:580] [field_sort] failed to parse field [order]\"}],\"type\":\"x_content_parse_exception\",\"reason\":\"[1:580] [field_sort] failed to parse field [order]\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"No enum constant org.elasticsearch.search.sort.SortOrder.A\"}},\"status\":400}"}
    at respond (/Users/tsullivan/elastic/kibana/node_modules/elasticsearch/src/lib/transport.js:315:15)
    at checkRespForFailure (/Users/tsullivan/elastic/kibana/node_modules/elasticsearch/src/lib/transport.js:274:7)
    at HttpConnector.<anonymous> (/Users/tsullivan/elastic/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:166:7)
    at IncomingMessage.wrapper (/Users/tsullivan/elastic/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)
    at IncomingMessage.emit (events.js:194:15)
    at endReadableNT (_stream_readable.js:1103:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)
server    log   [14:51:07.852] [error][reporting][savedobject-csv] Generate CSV Error! [x_content_parse_exception] [1:580] [field_sort] failed to parse field [order] :: {"path":"/babynames/_search","query":{"scroll":"30s","size":500},"body":"{\"_source\":{\"includes\":[\"date\",\"_source\"]},\"docvalue_fields\":[\"date\"],\"query\":{\"bool\":{\"must\":[],\"filter\":[{\"match_all\":{}},{\"range\":{\"date\":{\"format\":\"strict_date_time\",\"gte\":\"1974-01-01T00:00:00-07:00\",\"lte\":\"2019-08-13T14:51:07-07:00\"}}},{\"bool\":{\"must\":[],\"filter\":[{\"match_all\":{}},{\"match_all\":{}},{\"range\":{\"date\":{\"format\":\"strict_date_optional_time\",\"gte\":\"1974-01-01T07:00:00.000Z\",\"lte\":\"2019-08-13T21:51:07.742Z\"}}}],\"should\":[],\"must_not\":[]}}],\"should\":[],\"must_not\":[]}},\"script_fields\":{},\"sort\":[{\"date\":{\"order\":\"desc\",\"unmapped_type\":\"boolean\"}},{\"d\":{\"order\":\"a\"}},{\"d\":{\"order\":\"e\"}}]}","statusCode":400,"response":"{\"error\":{\"root_cause\":[{\"type\":\"x_content_parse_exception\",\"reason\":\"[1:580] [field_sort] failed to parse field [order]\"}],\"type\":\"x_content_parse_exception\",\"reason\":\"[1:580] [field_sort] failed to parse field [order]\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"No enum constant org.elasticsearch.search.sort.SortOrder.A\"}},\"status\":400}"}
server   error  [14:51:07.763]  [x_content_parse_exception] [1:580] [field_sort] failed to parse field [order] :: {"path":"/babynames/_search","query":{"scroll":"30s","size":500},"body":"{\"_source\":{\"includes\":[\"date\",\"_source\"]},\"docvalue_fields\":[\"date\"],\"query\":{\"bool\":{\"must\":[],\"filter\":[{\"match_all\":{}},{\"range\":{\"date\":{\"format\":\"strict_date_time\",\"gte\":\"1974-01-01T00:00:00-07:00\",\"lte\":\"2019-08-13T14:51:07-07:00\"}}},{\"bool\":{\"must\":[],\"filter\":[{\"match_all\":{}},{\"match_all\":{}},{\"range\":{\"date\":{\"format\":\"strict_date_optional_time\",\"gte\":\"1974-01-01T07:00:00.000Z\",\"lte\":\"2019-08-13T21:51:07.742Z\"}}}],\"should\":[],\"must_not\":[]}}],\"should\":[],\"must_not\":[]}},\"script_fields\":{},\"sort\":[{\"date\":{\"order\":\"desc\",\"unmapped_type\":\"boolean\"}},{\"d\":{\"order\":\"a\"}},{\"d\":{\"order\":\"e\"}}]}","statusCode":400,"response":"{\"error\":{\"root_cause\":[{\"type\":\"x_content_parse_exception\",\"reason\":\"[1:580] [field_sort] failed to parse field [order]\"}],\"type\":\"x_content_parse_exception\",\"reason\":\"[1:580] [field_sort] failed to parse field [order]\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"No enum constant org.elasticsearch.search.sort.SortOrder.A\"}},\"status\":400}"}
    at respond (/Users/tsullivan/elastic/kibana/node_modules/elasticsearch/src/lib/transport.js:315:15)
    at checkRespForFailure (/Users/tsullivan/elastic/kibana/node_modules/elasticsearch/src/lib/transport.js:274:7)
    at HttpConnector.<anonymous> (/Users/tsullivan/elastic/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:166:7)
    at IncomingMessage.wrapper (/Users/tsullivan/elastic/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)
    at IncomingMessage.emit (events.js:194:15)
    at endReadableNT (_stream_readable.js:1103:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)

[elasticmachine]

Pinging @elastic/kibana-stack-services

[bmcconaghy]

@tsullivan this should be fixed by #43038 (just merged last night), can you confirm?

[tsullivan]

Confirmed that @Bargs fixed this with the new migration

Thanks!

[tarkin67]

Hi @tsullivan

I am still experiencing this issue with migrated saved search panels in elastic & kibana 7.4.1:

Could you double check this is fixed? Our saved search has migration version 7.0.0:

{
    "id":  "d1f59d10-1891-11e9-a8fc-b99321995eaa",
    "type":  "search",
    "updated_at":  "2019-07-09T13:39:40.905Z",
    "version":  "WzQ4NSwyXQ==",
    "attributes":  {
                       "title":  "Wm6RollingFileAppender-Search",
                       "description":  "",
                       "hits":  0,
                       "columns":  [
                                       "level",
                                       "class",
                                       "thread.id",
                                       "thread.info",
                                       "logcontext",
                                       "message"
                                   ],
                       "sort":  [
                                    "@timestamp",
                                    "desc"
                                ],
                       "version":  1,
                       "kibanaSavedObjectMeta":  {
                                                     "searchSourceJSON":  "{\"highlightAll\":true,\"version\":true,\"query\":{\"language\":\"lucene\",\"query\":\"\"},\"filter\":[],\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.index\"}"
                                                 }
                   },
    "references":  [
                       {
                           "name":  "kibanaSavedObjectMeta.searchSourceJSON.index",
                           "id":  "cea14890-17fa-11e9-a8fc-b99321995eaa",
                           "type":  "index-pattern"
                       }
                   ],
    "migrationVersion":  {
                             "search":  "7.0.0"
                         }
}

Thanks and kind regards

EDIT:
This is the stacktrace in the Kibana console:

log   [09:13:08.765] [error][csv_from_savedobject][execute-job][immediate][reporting] Generate CSV Error! [x_content_parse_exception] [1:663] [field_sort] failed to parse field [order] :: {"path":"/filebeat-wm6rollingfileappender/_search","query":{"scroll":"30s","size":500},"body":"{\"_source\":{\"includes\":[\"@timestamp\",\"level\",\"class\",\"thread.id\",\"thread.info\",\"logcontext\",\"message\"]},\"docvalue_fields\":[\"@timestamp\"],\"query\":{\"bool\":{\"must\":[{\"match_all\":{}}],\"filter\":[{\"range\":{\"@timestamp\":{\"format\":\"strict_date_time\",\"gte\":\"2019-10-31T09:58:08+01:00\",\"lte\":\"2019-10-31T10:13:08+01:00\"}}},{\"bool\":{\"must\":[{\"match_all\":{}},{\"match_all\":{}}],\"filter\":[{\"range\":{\"@timestamp\":{\"format\":\"strict_date_optional_time\",\"gte\":\"2019-10-31T08:58:08.332Z\",\"lte\":\"2019-10-31T09:13:08.332Z\"}}}],\"should\":[],\"must_not\":[]}}],\"should\":[],\"must_not\":[]}},\"script_fields\":{},\"sort\":[{\"@timestamp\":{\"order\":\"desc\",\"unmapped_type\":\"boolean\"}},{\"@\":{\"order\":\"t\"}},{\"d\":{\"order\":\"e\"}}]}","statusCode":400,"response":"{\"error\":{\"root_cause\":[{\"type\":\"x_content_parse_exception\",\"reason\":\"[1:663] [field_sort] failed to parse field [order]\"}],\"type\":\"x_content_parse_exception\",\"reason\":\"[1:663] [field_sort] failed to parse field [order]\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"No enum constant org.elasticsearch.search.sort.SortOrder.T\"}},\"status\":400}"}
error  [09:13:08.684]  [x_content_parse_exception] [1:663] [field_sort] failed to parse field [order] :: {"path":"/filebeat-wm6rollingfileappender/_search","query":{"scroll":"30s","size":500},"body":"{\"_source\":{\"includes\":[\"@timestamp\",\"level\",\"class\",\"thread.id\",\"thread.info\",\"logcontext\",\"message\"]},\"docvalue_fields\":[\"@timestamp\"],\"query\":{\"bool\":{\"must\":[{\"match_all\":{}}],\"filter\":[{\"range\":{\"@timestamp\":{\"format\":\"strict_date_time\",\"gte\":\"2019-10-31T09:58:08+01:00\",\"lte\":\"2019-10-31T10:13:08+01:00\"}}},{\"bool\":{\"must\":[{\"match_all\":{}},{\"match_all\":{}}],\"filter\":[{\"range\":{\"@timestamp\":{\"format\":\"strict_date_optional_time\",\"gte\":\"2019-10-31T08:58:08.332Z\",\"lte\":\"2019-10-31T09:13:08.332Z\"}}}],\"should\":[],\"must_not\":[]}}],\"should\":[],\"must_not\":[]}},\"script_fields\":{},\"sort\":[{\"@timestamp\":{\"order\":\"desc\",\"unmapped_type\":\"boolean\"}},{\"@\":{\"order\":\"t\"}},{\"d\":{\"order\":\"e\"}}]}","statusCode":400,"response":"{\"error\":{\"root_cause\":[{\"type\":\"x_content_parse_exception\",\"reason\":\"[1:663] [field_sort] failed to parse field [order]\"}],\"type\":\"x_content_parse_exception\",\"reason\":\"[1:663] [field_sort] failed to parse field [order]\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"No enum constant org.elasticsearch.search.sort.SortOrder.T\"}},\"status\":400}"}
    at respond (C:\dev\Software\elastic\kibana-7.4.1-windows-x86_64\node_modules\elasticsearch\src\lib\transport.js:349:15)
    at checkRespForFailure (C:\dev\Software\elastic\kibana-7.4.1-windows-x86_64\node_modules\elasticsearch\src\lib\transport.js:306:7)
    at HttpConnector.<anonymous> (C:\dev\Software\elastic\kibana-7.4.1-windows-x86_64\node_modules\elasticsearch\src\lib\connectors\http.js:173:7)
    at IncomingMessage.wrapper (C:\dev\Software\elastic\kibana-7.4.1-windows-x86_64\node_modules\elasticsearch\node_modules\lodash\lodash.js:4929:19)
    at IncomingMessage.emit (events.js:194:15)
    at endReadableNT (_stream_readable.js:1103:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)