Network community_id processor for ingest pipelines

[danhermann]

Adds a processor that computes the community_id for flow data according to the Community ID Specification.

For example:

POST _ingest/pipeline/_simulate?verbose
{
  "pipeline": {
    "processors": [
      {
        "community_id": {
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "source": {
          "ip": "128.232.110.120",
          "port": 34855
        },
        "destination": {
          "ip": "66.35.250.204",
          "port": 80
        },
        "network": {
          "transport": "TCP"
        }
      }
    }
  ]
}

populates the network.community_id field as below:

...
"_source" : {
  "destination" : {
    "port" : 80,
    "ip" : "66.35.250.204"
  },
  "source" : {
    "port" : 34855,
    "ip" : "128.232.110.120"
  },
  "network" : {
    "community_id" : "1:LQU9qZlK+B5F3KDmev6m5PMibrg=",
    "transport" : "TCP"
  }
}
...

Closes #55685

[elasticmachine]

Pinging @elastic/es-core-features (Team:Core/Features)

[danhermann]

@andrewkroh, I would appreciate a look at this to see if it meets the criteria for #55685. I did replicate all the unit tests for the Beats community_id processor to verify that this processor produces the same values, so I'm reasonably confident about it in that regard.

[danhermann]

cc: @elastic/es-ui in case Kibana auto-complete needs to be updated with this new processor.

[alisonelizabeth]

Thanks @danhermann. I've opened elastic/kibana#86321 to track changes needed in Kibana. Can you also share the available options for this processor?

[danhermann]

Thanks @danhermann. I've opened elastic/kibana#86321 to track changes needed in Kibana. Can you also share the available options for this processor?

Yes, I'll open a separate PR with the docs for this processor that will include all its available options. I'll tag you on that one when I open it.

[andrewkroh]

This looks great. Just a minor comment from me.

[andrewkroh]

I'm worried that this could potentially perform network calls to lookup IP addresses from names.

Luckily we have a utility for this that's safe. Can you use that package here?

elasticsearch/server/src/main/java/org/elasticsearch/common/network/InetAddresses.java

Line 340 in feab123

public static InetAddress forString(String ipString) {

[andrewkroh]

Not sure if this is a worth it, so feel free to ignore. To ensure that fromNumber stays up-to-date with all the enum transportNumbers it might be useful to have test that checks the all the Transport.values() work with fromNumber without exception.

[danhermann]

@alisonelizabeth, the processor's options are documented here: #66592

[danhermann]

This looks great. Just a minor comment from me.

Thanks, @andrewkroh! I've made both changes that you suggested.

[andrewkroh]

LGTM

[martijnvg]

For processors added in ingest-common module there is a yaml test (Ingest common installed) that ensures that processors are correct wired up. Maybe we should have such a test for the xpack ingest module as well? Then we verify that uri_parts and community_id processors have correctly been wired up and can be used in pipelines.

Ingest integration LGTM.

[martijnvg]

make class final?

[danhermann]

@elasticmachine update branch

[danhermann]

Thanks, @martijnvg! I'll open another PR with the module yaml test that you suggested.