Update beats framework to 1304682

[graphaelli]

Pulls in ECS field definitions

[graphaelli]

The test failure is legitimate. https://github.com/elastic/kibana/blob/master/src/core_plugins/kibana/server/tutorials/apm/saved_objects/index_pattern.json does indeed have an error.code field as a number because as recently as 6.5 that's how libbeat defined it. With this libbeat update, the field has become a keyword - I assume this change is only intended for 7.0. Since we're about to discuss a strategy for handling these cases, I'm holding off resolving that.

@jalvz / @simitt While investigating this, I noticed the diff is only done in one direction. After checking #1168 I'm not sure if that's intentional. Applying this:

diff --git a/script/are_kibana_saved_objects_updated.py b/script/are_kibana_saved_objects_updated.py
index 98738c40..7ff1a985 100755
--- a/script/are_kibana_saved_objects_updated.py
+++ b/script/are_kibana_saved_objects_updated.py
@@ -81,7 +81,7 @@ def iterate(val_id, key, v1, v2):
             for item1, item2 in zip(v1, v2):
                 ret_val = max(ret_val, iterate(val_id, key, *json_val(item1, item2)))
     else:
-        d = jsondiff.diff(*json_val(v1, v2))
+        d = jsondiff.JsonDiffer(syntax='symmetric').diff(*json_val(v1, v2))
         if d:
             ret_val = 2
             print("Difference for id '{}' for key '{}'".format(val_id, key))

reveals some of our new fields missing on the kibana side:

---- Comparing Dashboards:
up-to-date
---- Comparing Index Pattern:
Difference for id 'apm-*' for key 'attributes.fields.name=timestamp.us'
[
    {
        "count": 1, 
        "name": "timestamp.us", 
        "searchable": true, 
        "analyzed": false, 
        "indexed": true, 
        "aggregatable": true, 
        "doc_values": true, 
        "type": "number", 
        "scripted": false
    }, 
    ""
]
Value in APM Server: {u'count': 1, u'name': u'timestamp.us', u'searchable': True, u'analyzed': False, u'indexed': True, u'aggregatable': True, u'doc_values': True, u'type': u'number', u'scripted': False}
Value in Kibana: 
---
Difference for id 'apm-*' for key 'attributes.fields.name=context.http.status_code'
[
    {
        "count": 0, 
        "name": "context.http.status_code", 
        "searchable": true, 
        "analyzed": false, 
        "indexed": true, 
        "aggregatable": true, 
        "doc_values": true, 
        "type": "number", 
        "scripted": false
    }, 
    ""
]
Value in APM Server: {u'count': 0, u'name': u'context.http.status_code', u'searchable': True, u'analyzed': False, u'indexed': True, u'aggregatable': True, u'doc_values': True, u'type': u'number', u'scripted': False}
Value in Kibana: 
---
Difference for id 'apm-*' for key 'attributes.fields.name=trace.id'
[
    {
        "count": 0, 
        "name": "trace.id", 
        "searchable": true, 
        "analyzed": false, 
        "indexed": true, 
        "aggregatable": true, 
        "doc_values": true, 
        "type": "string", 
        "scripted": false
    }, 
    ""
]
Value in APM Server: {u'count': 0, u'name': u'trace.id', u'searchable': True, u'analyzed': False, u'indexed': True, u'aggregatable': True, u'doc_values': True, u'type': u'string', u'scripted': False}
Value in Kibana: 
---
Difference for id 'apm-*' for key 'attributes.fields.name=parent.id'
[
    {
        "count": 0, 
        "name": "parent.id", 
        "searchable": true, 
        "analyzed": false, 
        "indexed": true, 
        "aggregatable": true, 
        "doc_values": true, 
        "type": "string", 
        "scripted": false
    }, 
    ""
]
Value in APM Server: {u'count': 0, u'name': u'parent.id', u'searchable': True, u'analyzed': False, u'indexed': True, u'aggregatable': True, u'doc_values': True, u'type': u'string', u'scripted': False}
Value in Kibana: 
---
Difference for id 'apm-*' for key 'attributes.fields.name=host.os.kernel'
[
    {
        "count": 0, 
        "name": "host.os.kernel", 
        "searchable": true, 
        "analyzed": false, 
        "indexed": true, 
        "aggregatable": true, 
        "doc_values": true, 
        "type": "string", 
        "scripted": false
    }, 
    ""
]
Value in APM Server: {u'count': 0, u'name': u'host.os.kernel', u'searchable': True, u'analyzed': False, u'indexed': True, u'aggregatable': True, u'doc_values': True, u'type': u'string', u'scripted': False}
Value in Kibana: 
---
Difference for id 'apm-*' for key 'attributes.fields.name=span.hex_id'
[
    {
        "count": 0, 
        "name": "span.hex_id", 
        "searchable": true, 
        "analyzed": false, 
        "indexed": true, 
        "aggregatable": true, 
        "doc_values": true, 
        "type": "string", 
        "scripted": false
    }, 
    ""
]
Value in APM Server: {u'count': 0, u'name': u'span.hex_id', u'searchable': True, u'analyzed': False, u'indexed': True, u'aggregatable': True, u'doc_values': True, u'type': u'string', u'scripted': False}
Value in Kibana: 
---
Difference for id 'apm-*' for key 'attributes.fields.name=span.sync'
[
    {
        "count": 0, 
        "name": "span.sync", 
        "searchable": true, 
        "analyzed": false, 
        "indexed": true, 
        "aggregatable": true, 
        "doc_values": true, 
        "scripted": false
    }, 
    ""
]
Value in APM Server: {u'count': 0, u'name': u'span.sync', u'searchable': True, u'analyzed': False, u'indexed': True, u'aggregatable': True, u'doc_values': True, u'scripted': False}
Value in Kibana: 
---
Difference for id 'apm-*' for key 'attributes.title'
[
    "apm-*", 
    ""
]
Value in APM Server: apm-*
Value in Kibana: 
---

It doesn't actually seem to cause a problem as index pattern loaded via the UI still had those fields, but I'm not sure why.

[simitt]

Thanks @graphaelli for figuring this out. It is indeed very confusing that the Kibana Index Pattern does have those values. I'll take a look at that.

[simitt]

#1571 and elastic/kibana#25929 are up to fix the Kibana index pattern issues.

[graphaelli]

Thanks for putting those in @simitt. elastic/beats#9188 is in for the other test failure - that is needed for overriding count on error.id in the kibana index pattern.

[graphaelli]

superseded by #1582