Low throughput rates for Cisco ASA module's Ingest Node pipeline

[andrewkroh]

There have been multiple reports of low event rates when using the Cisco ASA Filebeat module. In two independent analyses came to the conclusion was that one processor in the pipeline was taking a significant amount of time compared to the others. This was the date processor with timezone option.

beats/x-pack/filebeat/module/cisco/shared/ingest/asa-ftd-pipeline.yml

Lines 105 to 127 in 1da173a

	- date:
	if: "ctx.event.timezone != null"
	timezone: "{{ event.timezone }}"
	field: "_temp_.raw_date"
	target_field: "@timestamp"
	formats:
	- "ISO8601"
	- "MMM d HH:mm:ss"
	- "MMM dd HH:mm:ss"
	- "EEE MMM d HH:mm:ss"
	- "EEE MMM dd HH:mm:ss"
	- "MMM d HH:mm:ss z"
	- "MMM dd HH:mm:ss z"
	- "EEE MMM d HH:mm:ss z"
	- "EEE MMM dd HH:mm:ss z"
	- "MMM d yyyy HH:mm:ss"
	- "MMM dd yyyy HH:mm:ss"
	- "EEE MMM d yyyy HH:mm:ss"
	- "EEE MMM dd yyyy HH:mm:ss"
	- "MMM d yyyy HH:mm:ss z"
	- "MMM dd yyyy HH:mm:ss z"
	- "EEE MMM d yyyy HH:mm:ss z"
	- "EEE MMM dd yyyy HH:mm:ss z"

_nodes/stats metrics in one case showed that about a quarter of the processing time was spent in this processor (analyzed with https://github.com/andrewkroh/go-ingest-node-metrics). We need to investigate why this one processor takes more time and see if we can improve the overall throughput.

For confirmed bugs, please report:

• Version: 7.9, 7.10
• Relates: [Change Proposal] Pipeline benchmark test type package-spec#146

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[andrewkroh]

I think we need to setup a benchmark to replicate the issue. Then test a few changes and measure the results.

One change I'd like to see tested is what happens if the number of date patterns is reduced. And other whether including the timezone option affects the date processor's execution time.

[leehinman]

relates elastic/elasticsearch#73918

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!

[andrewkroh]

This date processor change probably affected performance under ES 7.17 and 8.x. We should retest.

elastic/elasticsearch#83764

[joegallo]

I'd expect elastic/elasticsearch#92880 to make a difference here, too.

[elasticmachine]

Pinging @elastic/sec-deployment-and-devices (Team:Security-Deployment and Devices)