[Filebeat][Fortinet Module] fortinet.firewall.addr is not always an IP address, causing ingest errors

[fredtj]

Hello,

See error message below:

2021-05-06T13:21:44.809+0100	WARN	[elasticsearch]	elasticsearch/client.go:408	Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xc01d15b27b75e4a8, ext:2855365583854, loc:(*time.Location)(0x637e5a0)}, Meta:{"pipeline":"filebeat-7.12.1-fortinet-firewall-pipeline","truncated":false}, Fields:{"agent":{"ephemeral_id":"6a006669-b93a-4213-8a67-6ee6ab7929b6","hostname":"xxx","id":"89ad49b4-bcd9-41c1-b4fa-2d004c365ec5","name":"xxx","type":"filebeat","version":"7.12.1"},"ecs":{"version":"1.8.0"},"event":{"dataset":"fortinet.firewall","module":"fortinet","timezone":"+01:00"},"fileset":{"name":"firewall"},"input":{"type":"udp"},"log":{"source":{"address":"xxx:41272"}},"message":"\u003c190\u003elogver=604051828 timestamp=1620303642 tz=\"UTC+1:00\" devname=\"xxx\" devid=\"xxx\" vd=\"root\" date=2021-05-06 time=13:20:42 eventtime=1620303643069933174 tz=\"+0100\" logid=\"0112053203\" type=\"event\" subtype=\"connector\" level=\"information\" logdesc=\"Dynamic address updated\" fctemssn=\"(null)\" addr=\"MAC_FCTEMS00000xxxxx_Bitlocker-Off\" msg=\"Updated tag MAC_FCTEMS00000xxxxx_Bitlocker-Off.\"\n","service":{"type":"fortinet"},"tags":["fortinet-firewall","forwarded"]}, Private:interface {}(nil), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [fortinet.firewall.addr] of type [ip] in document with id '-2GgQXkBW99wFEOBcg-8'. Preview of field's value: 'MAC_FCTEMS00000xxxxx_Bitlocker-Off'","caused_by":{"type":"illegal_argument_exception","reason":"'MAC_FCTEMS00000xxxxx_Bitlocker-Off' is not an IP string literal."}}

It seems when Fortigate updates a dynamic address book entry, the resulting log message spits out the object name and not an IP. This causes the above error and the document is not ingested.

I fixed this on my set up by adding the following to the firewall pipeline:

      {
        "rename" : {
          "field" : "fortinet.firewall.addr",
          "target_field" : "fortinet.firewall.addrgrp",
          "if" : "ctx.fortinet?.firewall?.logdesc == 'Dynamic address updated'",
          "ignore_missing" : true
        }
      },

I'm not sure if this is the correct approach, but it does seem to have resolved the issue and those logs are now ingested correctly.

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[legoguy1000]

There are no sample logs with this situation. I can make this change easily. @fredtj can you provide some sample logs that I can use in the testing to validate the changes??

[fredtj]

Use these logs for your testing and ignore the previous sample, as that had extra fields added by FortiAnalyzer, which is not supported by this module yet.

<190>devname="firewall" devid="FG201EEF34CD12AB" vd="root" date=2021-05-07 time=08:31:14 eventtime=1620372674880370858 tz="+0100" logid="0112053203" type="event" subtype="connector" level="information" logdesc="Dynamic address updated" fctemssn="FCTEMS0000011111" addr="FCTEMS0000011111_AV-Running" msg="Updated tag FCTEMS0000011111_AV-Running."
<190>devname="firewall" devid="FG201EEF34CD12AB" vd="root" date=2021-05-07 time=08:31:14 eventtime=1620372674880455433 tz="+0100" logid="0112053203" type="event" subtype="connector" level="information" logdesc="Dynamic address updated" fctemssn="FCTEMS0000011111" addr="MAC_FCTEMS0000011111_AV-Running" msg="Updated tag MAC_FCTEMS0000011111_AV-Running."
<190>devname="firewall" devid="FG201EEF34CD12AB" vd="root" date=2021-05-07 time=08:31:14 eventtime=1620372674880744919 tz="+0100" logid="0112053203" type="event" subtype="connector" level="information" logdesc="Dynamic address updated" fctemssn="FCTEMS0000011111" addr="FCTEMS0000011111_Connected-to-EMS" msg="Updated tag FCTEMS0000011111_Connected-to-EMS."
<190>devname="firewall" devid="FG201EEF34CD12AB" vd="root" date=2021-05-07 time=08:31:14 eventtime=1620372674880784143 tz="+0100" logid="0112053203" type="event" subtype="connector" level="information" logdesc="Dynamic address updated" fctemssn="FCTEMS0000011111" addr="MAC_FCTEMS0000011111_Connected-to-EMS" msg="Updated tag MAC_FCTEMS0000011111_Connected-to-EMS."
<190>devname="firewall" devid="FG201EAB12CD34EF" vd="root" date=2021-05-07 time=08:31:14 eventtime=1620372674900027938 tz="+0100" logid="0112053203" type="event" subtype="connector" level="information" logdesc="Dynamic address updated" fctemssn="(null)" addr="FCTEMS0000011111_AV-Running" msg="Updated tag FCTEMS0000011111_AV-Running."
<190>devname="firewall" devid="FG201EAB12CD34EF" vd="root" date=2021-05-07 time=08:31:14 eventtime=1620372674900167367 tz="+0100" logid="0112053203" type="event" subtype="connector" level="information" logdesc="Dynamic address updated" fctemssn="(null)" addr="MAC_FCTEMS0000011111_AV-Running" msg="Updated tag MAC_FCTEMS0000011111_AV-Running."
<190>devname="firewall" devid="FG201EAB12CD34EF" vd="root" date=2021-05-07 time=08:31:14 eventtime=1620372674900749585 tz="+0100" logid="0112053203" type="event" subtype="connector" level="information" logdesc="Dynamic address updated" fctemssn="(null)" addr="FCTEMS0000011111_Connected-to-EMS" msg="Updated tag FCTEMS0000011111_Connected-to-EMS."
<190>devname="firewall" devid="FG201EAB12CD34EF" vd="root" date=2021-05-07 time=08:31:14 eventtime=1620372674900961834 tz="+0100" logid="0112053203" type="event" subtype="connector" level="information" logdesc="Dynamic address updated" fctemssn="(null)" addr="MAC_FCTEMS0000011111_Connected-to-EMS" msg="Updated tag MAC_FCTEMS0000011111_Connected-to-EMS."

[legoguy1000]

Also if u want to make a separate issue for the fortiAnalyzer, I'd be happy to work on that with u, either updating the existing filesets or create a new one.

[fredtj]

@legoguy1000 see here: #19315

[legoguy1000]

@fredtj PR is ready for review. If you have any comments, let me know.