Default ouput of docvalue_fields will soon change

[jpountz]

Hello Kibana team,

I'm reaching out because I'm about to merge a pull request to master (7.0) that changes the default format of docvalue_fields: elastic/elasticsearch#30831. This is mostly a change for date fields, which will now be formatted based on the format that is configured in the mappings. For instance if you would like to get a number of millis since Epoch instead, you should now pass something like docvalue_fields: [ { "field": "@timestamp", "format": "epoch_millis" } ].

I see that @Bargs already did a change to add format: use_field_mapping some months ago #22771. Does Kibana always specify a format when fetching docvalue_fields? If this isn't the case, then the Elasticsearch PR might break Kibana, which I'd like to avoid.

[elasticmachine]

Pinging @elastic/kibana-app

[timroes]

@joelgriffith @tsullivan it seems that beside what Matt already fixed, we still have two occurrences of docvalue_fields in some reporting tests: https://github.com/elastic/kibana/blob/master/x-pack/test/reporting/api/generation_urls.js#L19

Could you please check if that means some reporting tests will break due to that change?

[joelgriffith]

Yup, will check and report back.

[jpountz]

@joelgriffith Let me know if I can help.

[joelgriffith]

@jpountz I'm taking a look now and seeing what I can find. Is there a snapshot of this version of ES I can run to do a quick smoke test (I'm not fully setup to develop ES locally, but also not sure how hard it is to get a dev build published someplace)?

[jpountz]

@joelgriffith I can build one for you. Would you rather like the zip or tar.gz distribution?

[joelgriffith]

Thanks! Doesn't matter, either one :)

[joelgriffith]

Sorry for my sluggish response, I had a few things to get up to speed on :)

So, it appears the prior URL's were of format:
docvalue_fields: ['@timestamp']. Judging by your comment I assume these are no longer valid as they can be ambiguously interpreted?

The other concern that @tsullivan pointed out is that saved/generated reports may have this value hard-coded as well, which (if so) represents another set of challenges to getting this pushed through.

[jpountz]

So, it appears the prior URL's were of format:
docvalue_fields: ['@timestamp']. Judging by your comment I assume these are no longer valid as they can be ambiguously interpreted?

This is still valid, but we are changing the return value from a long to a string, which is the same date formatted with the format that is configured in the mappings. If you need to get the number of millis since Epoch, you can still do that by passing a custom format: docvalue_fields: [{'field': '@timestamp', 'format': 'epoch_millis'}]. This will return the number of millis since Epoch (as a string).

6.x already supports opting in for the 7.x behavior by doing docvalue_fields: [{'field': '@timestamp', 'format': 'use_field_mapping'}] but this will become deprecated in 7.x.

[joelgriffith]

I was able to run reporting tests locally (after some back and forth) with passes using the branch mentioned here! I think we're good to go with this:

$ node ./scripts/functional_test_runner.js --config=x-pack/test/reporting/configs/chromium_api.js --grep='reporting usage' info Config loaded
 info Starting tests

 └-: chromium
   └-> "before all" hook
   └-> "before all" hook
     │ info [/Users/joelgriffith/Projects/kibana/test/functional/fixtures/es_archiver/dashboard/current/kibana] Loading "mappings.json"
     │ info [/Users/joelgriffith/Projects/kibana/test/functional/fixtures/es_archiver/dashboard/current/kibana] Loading "data.json.gz"
     │ info [/Users/joelgriffith/Projects/kibana/test/functional/fixtures/es_archiver/dashboard/current/kibana] Deleted existing index [".kibana_2",".kibana_1"]
     │ info [/Users/joelgriffith/Projects/kibana/test/functional/fixtures/es_archiver/dashboard/current/kibana] Created index ".kibana"
     │ info [/Users/joelgriffith/Projects/kibana/test/functional/fixtures/es_archiver/dashboard/current/kibana] Indexed 84 docs into ".kibana"
     │ info [migrations] Creating index .kibana_2.
     │ info [migrations] Reindexing .kibana to .kibana_1
     │ info [migrations] Migrating .kibana_1 saved objects to .kibana_2
     │ info [migrations] Pointing alias .kibana to .kibana_2.
     │ info [migrations] Finished in 995ms.
     │ info [/Users/joelgriffith/Projects/kibana/test/functional/fixtures/es_archiver/dashboard/current/data] Loading "mappings.json"
     │ info [/Users/joelgriffith/Projects/kibana/test/functional/fixtures/es_archiver/dashboard/current/data] Loading "data.json.gz"
     │ info [/Users/joelgriffith/Projects/kibana/test/functional/fixtures/es_archiver/dashboard/current/data] Created index "animals-dogs-2018-01-01"
     │ info [/Users/joelgriffith/Projects/kibana/test/functional/fixtures/es_archiver/dashboard/current/data] Created index "animals-dogs-2018-04-10"
     │ info [/Users/joelgriffith/Projects/kibana/test/functional/fixtures/es_archiver/dashboard/current/data] Created index "animals-cats-2018-01-01"
     │ info [/Users/joelgriffith/Projects/kibana/test/functional/fixtures/es_archiver/dashboard/current/data] Created index "animals-cats-2018-04-10"
     │ info [/Users/joelgriffith/Projects/kibana/test/functional/fixtures/es_archiver/dashboard/current/data] Created index "dogbreeds"
     │ info [/Users/joelgriffith/Projects/kibana/test/functional/fixtures/es_archiver/dashboard/current/data] Created index "logstash-0"
     │ info [/Users/joelgriffith/Projects/kibana/test/functional/fixtures/es_archiver/dashboard/current/data] Indexed 6 docs into "animals-dogs-2018-01-01"
     │ info [/Users/joelgriffith/Projects/kibana/test/functional/fixtures/es_archiver/dashboard/current/data] Indexed 6 docs into "animals-dogs-2018-04-10"
     │ info [/Users/joelgriffith/Projects/kibana/test/functional/fixtures/es_archiver/dashboard/current/data] Indexed 6 docs into "animals-cats-2018-01-01"
     │ info [/Users/joelgriffith/Projects/kibana/test/functional/fixtures/es_archiver/dashboard/current/data] Indexed 5 docs into "animals-cats-2018-04-10"
     │ info [/Users/joelgriffith/Projects/kibana/test/functional/fixtures/es_archiver/dashboard/current/data] Indexed 6 docs into "dogbreeds"
     │ info [/Users/joelgriffith/Projects/kibana/test/functional/fixtures/es_archiver/dashboard/current/data] Indexed 14005 docs into "logstash-0"
   └-: reporting usage
     └-> "before all" hook
     └-> "before all" hook
     └-: initial state
       └-> "before all" hook
       └-> "before all" hook
       └-> shows reporting as available and enabled
         └-> "before each" hook: global before each
         └- ✓ pass
       └-> "after each" hook
       └-> all counts are 0
         └-> "before each" hook: global before each
         └- ✓ pass
       └-> "after each" hook
       └-> "after all" hook
     └-: from archive data
       └-> "before all" hook
       └-> generated from 6.2
         └-> "before each" hook: global before each
         │ info [bwc/6_2] Loading "mappings.json"
         │ info [bwc/6_2] Loading "data.json.gz"
         │ info [bwc/6_2] Created index ".reporting-2018.03.11"
         │ info [bwc/6_2] Indexed 8 docs into ".reporting-2018.03.11"
         └- ✓ pass
       └-> "after each" hook
       └-> generated from 6.3
         └-> "before each" hook: global before each
         │ info [bwc/6_3] Loading "mappings.json"
         │ info [bwc/6_3] Loading "data.json.gz"
         │ info [bwc/6_3] Created index ".reporting-2018.05.06"
         │ info [bwc/6_3] Indexed 14 docs into ".reporting-2018.05.06"
         └- ✓ pass
       └-> "after each" hook
       └-> "after all" hook
     └-: from new jobs posted
       └-> "before all" hook
       └-> csv
         └-> "before each" hook: global before each
         └- ✓ pass
       └-> "after each" hook
       └-> preserve_layout pdf
         └-> "before each" hook: global before each
         └- ✓ pass  (18.3s)
       └-> "after each" hook
       └-> print_layout pdf
         └-> "before each" hook: global before each
         └- ✓ pass  (16.8s)
       └-> "after each" hook
       └-> "after all" hook
     └-> "after all" hook
   └-> "after all" hook
     │ info [/Users/joelgriffith/Projects/kibana/test/functional/fixtures/es_archiver/dashboard/current/kibana] Unloading indices from "mappings.json"
     │ warn since spaces are enabled, all objects other than the default space were deleted from .kibana rather than deleting the whole index
     │ info [/Users/joelgriffith/Projects/kibana/test/functional/fixtures/es_archiver/dashboard/current/kibana] Deleted existing index ".kibana"
     │ info [/Users/joelgriffith/Projects/kibana/test/functional/fixtures/es_archiver/dashboard/current/kibana] Unloading indices from "data.json.gz"
     │ info [/Users/joelgriffith/Projects/kibana/test/functional/fixtures/es_archiver/dashboard/current/data] Unloading indices from "mappings.json"
     │ info [/Users/joelgriffith/Projects/kibana/test/functional/fixtures/es_archiver/dashboard/current/data] Deleted existing index "animals-dogs-2018-01-01"
     │ info [/Users/joelgriffith/Projects/kibana/test/functional/fixtures/es_archiver/dashboard/current/data] Deleted existing index "animals-dogs-2018-04-10"
     │ info [/Users/joelgriffith/Projects/kibana/test/functional/fixtures/es_archiver/dashboard/current/data] Deleted existing index "animals-cats-2018-01-01"
     │ info [/Users/joelgriffith/Projects/kibana/test/functional/fixtures/es_archiver/dashboard/current/data] Deleted existing index "animals-cats-2018-04-10"
     │ info [/Users/joelgriffith/Projects/kibana/test/functional/fixtures/es_archiver/dashboard/current/data] Deleted existing index "dogbreeds"
     │ info [/Users/joelgriffith/Projects/kibana/test/functional/fixtures/es_archiver/dashboard/current/data] Deleted existing index "logstash-0"
     │ info [/Users/joelgriffith/Projects/kibana/test/functional/fixtures/es_archiver/dashboard/current/data] Unloading indices from "data.json.gz"
   └-> "after all" hook


7 passing (1.0m)

[jpountz]

Thanks @joelgriffith for checking, I'll close this issue then.