[8.15](backport #40699) Add asynchronous ACK handling to S3 and SQS inputs

[mergify]

Modify SQS ingestion to listen for ACKs asynchronously so that input workers can keep reading new objects after a previous one has been published, instead of blocking on full upstream ingestion. This addresses the bottleneck where ingesting many small objects is slow as each one waits for a full ingestion round trip.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

This can best be tested by ingesting data from a live S3 or SQS queue. The scenario that most highlights the changed performance is ingesting many small individual objects.

Related issues

• Closes aws-s3 input workers shouldn't wait for objects to be fully ingested before starting the next object #39414

  ---

  This is an automatic backport of pull request Add asynchronous ACK handling to S3 and SQS inputs #40699 done by Mergify.

[mergify]

Cherry-pick of d2867fd has failed:

On branch mergify/bp/8.15/pr-40699
Your branch is up to date with 'origin/8.15'.

You are currently cherry-picking commit d2867fdd9f.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   CHANGELOG.next.asciidoc
	modified:   NOTICE.txt
	modified:   go.mod
	modified:   x-pack/filebeat/_meta/config/filebeat.inputs.reference.xpack.yml.tmpl
	modified:   x-pack/filebeat/docs/inputs/input-aws-s3.asciidoc
	modified:   x-pack/filebeat/filebeat.reference.yml
	new file:   x-pack/filebeat/input/awss3/acks.go
	modified:   x-pack/filebeat/input/awss3/config.go
	modified:   x-pack/filebeat/input/awss3/config_test.go
	modified:   x-pack/filebeat/input/awss3/input_integration_test.go
	modified:   x-pack/filebeat/input/awss3/interfaces.go
	modified:   x-pack/filebeat/input/awss3/mock_interfaces_test.go
	modified:   x-pack/filebeat/input/awss3/s3.go
	modified:   x-pack/filebeat/input/awss3/s3_input.go
	modified:   x-pack/filebeat/input/awss3/s3_objects_test.go
	modified:   x-pack/filebeat/input/awss3/s3_test.go
	modified:   x-pack/filebeat/input/awss3/sqs_input.go
	modified:   x-pack/filebeat/input/awss3/sqs_s3_event.go
	modified:   x-pack/filebeat/input/awss3/sqs_test.go
	modified:   x-pack/filebeat/module/aws/_meta/config.yml
	modified:   x-pack/filebeat/module/aws/cloudtrail/config/aws-s3.yml
	modified:   x-pack/filebeat/module/aws/cloudtrail/manifest.yml
	modified:   x-pack/filebeat/module/aws/s3access/config/aws-s3.yml
	modified:   x-pack/filebeat/module/aws/s3access/manifest.yml
	modified:   x-pack/filebeat/module/aws/vpcflow/config/input.yml
	modified:   x-pack/filebeat/module/aws/vpcflow/manifest.yml
	modified:   x-pack/filebeat/modules.d/aws.yml.disabled

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   go.sum
	both modified:   x-pack/filebeat/input/awss3/input_benchmark_test.go
	both modified:   x-pack/filebeat/input/awss3/s3_objects.go
	both modified:   x-pack/filebeat/input/awss3/sqs_s3_event_test.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[botelastic]

This pull request doesn't have a Team:<team> label.

[pierrehilbert]

Not sure we should backport this to 8.15.
cc @cmacknz / @nimarezainia / @flexitrev

[cmacknz]

When/if we merge the backport is more a question of how long we want it to soak or if we want to do additional testing before release.

This is a performance bugfix not a new feature so at a high level we could backport it.

[nimarezainia]

Not sure we should backport this to 8.15. cc @cmacknz / @nimarezainia / @flexitrev

Agreed. it's touching a lot of files and not obvious that a user is in anyway blocked. I suggest avoid backport as long as possible.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b mergify/bp/8.15/pr-40699 upstream/mergify/bp/8.15/pr-40699
git merge upstream/8.15
git push upstream mergify/bp/8.15/pr-40699