[Filebeat] Add option for S3 input to work without SQS notification

[kaiyan-sheng]

Current S3 input in Filebeat relies on SQS to send notification when a new log message is stored in S3 bucket. Some users only want a simple polling method applied on the S3 bucket periodically without SQS, similar to Logstash s3 input .

This will be very useful especially for users who don't need near real-time log message collection. The setup will be a lot simpler without SQS involved.

This idea is brought up in discuss forum issue https://discuss.elastic.co/t/aws-vpcflow-errors-count-not-find-region-configuration-context-deadline-exceeded.

[elasticmachine]

Pinging @elastic/integrations-platforms (Team:Platforms)

[kaiyan-sheng]

@mukeshelastic I would love to get your opinion on this. Thank you!!

[kaiyan-sheng]

Note: If we decide to implement this, then the current s3 input probably will be renamed to s3-sqs input instead.

[kaiyan-sheng]

We might have to wait till #15324 is done before start working on this one.

[jamiehynds]

@kaiyan-sheng we recently shipped a Cisco Umbrella integration which leverages the Beats S3 input. This works fine for Umbrella customers using self-managed S3 buckets, but Cisco also provide Cisco-managed buckets where SQS notifications are not available. It looks like we don't be able to support Cisco-managed buckets until this issue is address.

/cc @P1llus

[P1llus]

There is certainly needs and usecases for this indeed! Custom log sources could be fetched with Logstash, but if we want to build modules that store their data on a S3 bucket without a SQS queue we do indeed need that option.

[aspacca]

@kaiyan-sheng

Note: If we decide to implement this, then the current s3 input probably will be renamed to s3-sqs input instead.

Do we need a different input or can we use the same input relying either queue_url or s3_bucket is set in the config?

[kaiyan-sheng]

@aspacca I prefer using the same input aws-s3 and just adding a new config. WDYT?

[aspacca]

@kaiyan-sheng totally agree

[LaZyDK]

This will allow Filebeat to pull Cisco Umbrella logs from a Cisco Managed S3 bucket. This is also possible with Logstash today.