Mapping error ingesting logstash loggin

[psanz-estc]

ES returns a mapping error when trying to ingest Logstash logs (in json format) via logstash module

FB mapping:

 "logstash": {
          "properties": {
            "log": {
              "properties": {
                "log_event": {
                  "type": "object"
                },
                "module": {
                  "type": "keyword",
                  "ignore_above": 1024
                },
                "pipeline_id": {
                  "type": "keyword",
                  "ignore_above": 1024
                },
                "thread": {
                  "type": "keyword",
                  "fields": {
                    "text": {
                      "type": "text",
                      "norms": false
                    }
                  },
                  "ignore_above": 1024
                }
              }
            },

Event in Filebeat:

2020-08-14T17:52:51.518+0100	DEBUG	[processors]	processing/processors.go:186	Publish event: {
  "@timestamp": "2020-08-14T16:52:51.518Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.5.2",
    "pipeline": "filebeat-7.5.2-logstash-log-pipeline-json"
  },
  "input": {
    "type": "log"
  },
  "event": {
    "module": "logstash",
    "dataset": "logstash.log"
  },
  "ecs": {
    "version": "1.1.0"
  },
  "host": {
    "containerized": false,
    "hostname": "indexXXXX",
    "architecture": "x86_64",
    "os": {
      "kernel": "3.10.0-1062.7.1.el7.x86_64",
      "codename": "Maipo",
      "platform": "rhel",
      "version": "7.7 (Maipo)",
      "family": "redhat",
      "name": "Red Hat Enterprise Linux Server"
    },
    "name": "indexXXXX",
    "id": "2d268dd2fe52485fbac908a2d7eb4ec7"
  },
  "message": "{\"level\":\"WARN\",\"loggerName\":\"logstash.outputs.elasticsearch\",\"timeMillis\":1597423971116,\"thread\":\"[XXXXX-logs]>worker0\",\"logEvent\":{\"message\":\"Could not index event to Elasticsearch.\",\"status\":400,\"action\":[\"index\",{\"_index\":\"indexXXXX\",\"_type\":\"_doc\"},{\"metaClass\":{\"metaClass\":{\"metaClass\":{\"action\":\"[\\\"index\\\", {:_id=>nil, :_index=>\\\"indexXXXX\\\", :routing=>nil, :_type=>\\\"_doc\\\"}, #<LogStash::Event:0x2ee7e327>]\",\"response\":{\"index\":{\"_index\":\"indexXXXX-2020.08.13-000001\",\"_type\":\"_doc\",\"_id\":\"dNXj7XMB12EK1s8HakX7\",\"status\":400,\"error\":{\"type\":\"strict_dynamic_mapping_exception\",\"reason\":\"mapping set to strict, dynamic introduction of [guitester] within [_doc] is not allowed\"}}}}}}}]}}",
  "fileset": {
    "name": "log"
  },
  "agent": {
    "ephemeral_id": "8c03cbf1-4e9e-4d49-8929-a35e34313565",
    "hostname": "XXXXXXX",
    "id": "133b094d-2331-4159-908e-f2b9287a5fa7",
    "version": "7.5.2",
    "type": "filebeat"
  },
  "log": {
    "offset": 2179,
    "file": {
      "path": "/var/logs/logs/logstash-json.log"
    }
  },
  "service": {
    "type": "logstash"
  }
}

And it is rejected in ES due the mapping not being aligned:

2020-08-14T17:52:52.525+0100	WARN	elasticsearch/client.go:511	Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfc5ce38dee4ad4e, ext:101083374145, loc:(*time.Location)(0x4e5d700)}, Meta:common.MapStr{"pipeline":"filebeat-7.5.2-logstash-log-pipeline-json"}, Fields:common.MapStr{"agent":common.MapStr{"ephemeral_id":"8c03cbf1-4e9e-4d49-8929-a35e34313565", "hostname":"XXXXXXX", "id":"133b094d-2331-4159-908e-f2b9287a5fa7", "type":"filebeat", "version":"7.5.2"}, "ecs":common.MapStr{"version":"1.1.0"}, "event":common.MapStr{"dataset":"logstash.log", "module":"logstash"}, "fileset":common.MapStr{"name":"log"}, "host":common.MapStr{"architecture":"x86_64", "containerized":false, "hostname":"XXXXXXX", "id":"2d268dd2fe52485fbac908a2d7eb4ec7", "name":"XXXXXXX", "os":common.MapStr{"codename":"Maipo", "family":"redhat", "kernel":"3.10.0-1062.7.1.el7.x86_64", "name":"Red Hat Enterprise Linux Server", "platform":"rhel", "version":"7.7 (Maipo)"}}, "input":common.MapStr{"type":"log"}, "log":common.MapStr{"file":common.MapStr{"path":"/var/logs/logs/logstash-json.log"}, "offset":2179}, "message":"{\"level\":\"WARN\",\"loggerName\":\"logstash.outputs.elasticsearch\",\"timeMillis\":1597423971116,\"thread\":\"[indexXXXX]>worker0\",\"logEvent\":{\"message\":\"Could not index event to Elasticsearch.\",\"status\":400,\"action\":[\"index\",{\"_index\":\"indexXXXX\",\"_type\":\"_doc\"},{\"metaClass\":{\"metaClass\":{\"metaClass\":{\"action\":\"[\\\"index\\\", {:_id=>nil, :_index=>\\\"indexXXXX\\\", :routing=>nil, :_type=>\\\"_doc\\\"}, #<LogStash::Event:0x2ee7e327>]\",\"response\":{\"index\":{\"_index\":\"indexXXXX-2020.08.13-000001\",\"_type\":\"_doc\",\"_id\":\"dNXj7XMB12EK1s8HakX7\",\"status\":400,\"error\":{\"type\":\"strict_dynamic_mapping_exception\",\"reason\":\"mapping set to strict, dynamic introduction of [guitester] within [_doc] is not allowed\"}}}}}}}]}}", "service":common.MapStr{"type":"logstash"}}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc000443a00), Source:"/var/logs/log/logstash-json.log", Offset:2904, Timestamp:time.Time{wall:0xbfc5ce389ed00996, ext:100082021520, loc:(*time.Location)(0x4e5d700)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x4f, Device:0xfd06}}, TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"illegal_argument_exception","reason":"mapper [logstash.log.log_event.action] of different type, current_type [keyword], merged_type [ObjectMapper]"}

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[andrewkroh]

Here's another log sample (in pretty format) that causes the bug:

{
  "level": "WARN",
  "loggerName": "logstash.outputs.elasticsearch",
  "timeMillis": 1612827484046,
  "thread": "[foo]>worker1",
  "logEvent": {
    "message": "Could not index event to Elasticsearch.",
    "status": 400,
    "action": [
      "update",
      {
        "_id": "foo-1234abcd-96c6-4828-bcd4-51d33a156431",
        "_index": "filebeat-foo-2021.02",
        "_type": "_doc",
        "retry_on_conflict": 1
      },
      {
        "metaClass": {
          "metaClass": {
            "metaClass": {
              "action": "[\"update\", {:_id=>\"foo-1234abcd-96c6-4828-bcd4-51d33a156431\", :_index=>\"filebeat-foo-2021.02\", :routing=>nil, :_type=>\"_doc\", :retry_on_conflict=>1}, #<LogStash::Event:0x51b7cd8e>]",
              "response": {
                "update": {
                  "_index": "filebeat-foo-2021.02",
                  "_type": "_doc",
                  "_id": "foo-1234abcd-96c6-4828-bcd4-51d33a156431",
                  "status": 400,
                  "error": {
                    "type": "mapper_parsing_exception",
                    "reason": "failed to parse field [bar] of type [long] in document with id 'foo-1234abcd-96c6-4828-bcd4-51d33a156431'. Preview of field's value: 'ABCDEFGHIJ'",
                    "caused_by": {
                      "type": "illegal_argument_exception",
                      "reason": "For input string: \"ABCDEFGHIJ\""
                    }
                  }
                }
              }
            }
          }
        }
      }
    ]
  }
}

It cannot use flattened as the data type for the logstash.log.log_event.action field since this module is OSS and therefore cannot depend on an Elastic licensed feature. So I think that fields.yml should add a mapping for the field and then in the ingest node pipeline enforce that the value (or each value in case of an array) is a string. If the value is not a string then take some action (remove it or "stringify" it).

[andrewkroh]

I implemented what I discussed in the previous post with #23944.