Extend PANW/panos module with globalprotect and hipmatch datasets

[adriansr]

We've received the following contribution from @felix-lessoer

Here's a pipeline. It needs ECS mappings:

PUT _ingest/pipeline/panw-globalprotect-extension
{
  "processors": [
    {
      "csv": {
        "field": "log.original",
        "target_fields": [
          "panw.panos.FUTUE_USE",
          "panw.panos.Receive Time",
          "panw.panos.Serial Number",
          "panw.panos.Type",
          "panw.panos.Threat/Content Type",
          "panw.panos.FUTURE_USE",
          "panw.panos.Generated Time",
          "panw.panos.Virtual System",
          "panw.panos.Event ID",
          "panw.panos.Stage",
          "panw.panos.Authentication Method",
          "panw.panos.Tunnel Type",
          "panw.panos.Source User",
          "panw.panos.Source Region",
          "panw.panos.Machine Name",
          "panw.panos.Public IP",
          "panw.panos.Public IPv6",
          "panw.panos.Private IP",
          "panw.panos.Private IPv6",
          "panw.panos.Host ID",
          "panw.panos.Serial Number2",
          "panw.panos.Client Version",
          "panw.panos.Client OS",
          "panw.panos.Client OS Version",
          "panw.panos.Repeat Count",
          "panw.panos.Reason",
          "panw.panos.Error",
          "panw.panos.Description",
          "panw.panos.Status",
          "panw.panos.Location",
          "panw.panos.Login Duration",
          "panw.panos.Connect Method",
          "panw.panos.Error Code",
          "panw.panos.Portal",
          "panw.panos.Sequence Number",
          "panw.panos.Action Flags",
          "panw.panos.High Res Timestamp",
          "panw.panos.Selection Type",
          "panw.panos.Response Time",
          "panw.panos.Priority",
          "panw.panos.Attempted Gateways",
          "panw.panos.Gateway"
        ],
        "trim": true,
        "if": "ctx.panw?.panos?.type == \"GLOBALPROTECT\""
      }
    },
    {
      "csv": {
        "field": "log.original",
        "target_fields": [
          "panw.panos.FUTURE_USE",
          "panw.panos.Receive Time",
          "panw.panos.Serial Number",
          "panw.panos.Type",
          "panw.panos.Threat/Content Type",
          "panw.panos.FUTURE_USE",
          "panw.panos.Generated Time",
          "panw.panos.Source User",
          "panw.panos.Virtual System",
          "panw.panos.Machine Name",
          "panw.panos.Operating System",
          "panw.panos.Source Address",
          "panw.panos.HIP",
          "panw.panos.Repeat Count",
          "panw.panos.HIP Type",
          "panw.panos.FUTURE_USE",
          "panw.panos.FUTURE_USE",
          "panw.panos.Sequence Number",
          "panw.panos.Action Flags",
          "panw.panos.Device Group Hierarchy Level 1",
          "panw.panos.Device Group Hierarchy Level 2",
          "panw.panos.Device Group Hierarchy Level 3",
          "panw.panos.Device Group Hierarchy Level 4",
          "panw.panos.Virtual System Name",
          "panw.panos.Device Name",
          "panw.panos.Virtual System ID",
          "panw.panos.IPv6 Source Address",
          "panw.panos.Host ID",
          "panw.panos.User Device Serial Number",
          "panw.panos.Device MAC Address",
          "panw.panos.High Resolution Timestamp"
        ],
        "trim": true,
        "if": "ctx.panw?.panos?.type == \"HIPMATCH\""
      }
    }
  ]
}

A few sample anonymized logs https://drive.google.com/file/d/18ouI1X9ZTF2EiQX_hTUbwSTCrqDoy0BC/view?usp=sharing

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[felix-lessoer]

Thanks @adriansr .
This pipeline was build to run after the current (v 0.5.0) panos module integration.
So it looks at log.original field and parses the CSV content.
You can find the documentation here: https://docs.paloaltonetworks.com/pan-os/10-0/pan-os-admin/monitoring/use-syslog-for-monitoring/syslog-field-descriptions/globalprotect-log-fields.html

The other log types can be build with the same schema. So we can extend panos to achieve complete coverage.
I can also organize sample logs for the other log types.

Let me know how you would like to proceed.

[legoguy1000]

I'm currently working the global protect and no user id log types as part of #24724 and #24722. I can try to add the hipmatch log types too.

[legoguy1000]

@felix-lessoer can you provide additional HIP Match logs. I have built the pipeline and only have 1 sample to test.

[legoguy1000]

PR opened, let me know if there are any changes needed.