Missing backup and delete for AWS S3 polling mode

[kaiyan-sheng]

Proposed commit message

When running Filebeat s3 input polling mode, we suppose to have the options to backup s3 objects into a specific bucket and delete the object after backup.

filebeat.inputs:
- type: aws-s3
  enabled: true
  credential_profile_name: elastic-observability
  bucket_arn: arn:aws:s3:::test-s3-bucket-ks
  default_region: eu-central-1
  backup_to_bucket_arn: arn:aws:s3:::test-s3-bucket-ks-backup
  delete_after_backup: true
  number_of_workers: 5

With this config, objects/logs will be ingested from the bucket specified in bucket_arn into Elasticsearch just fine but no backup or delete happened.

This PR is add the feature back into S3 polling mode when user specifies a backup bucket and/or delete after backup.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

With the same config, after adding this change, rebuild filebeat and we can see the new log log7 got ingested into Elasticsearch and after ingestion, this file is deleted from the test-s3-bucket-ks bucket and copied to test-s3-bucket-ks-backup bucket.

Related issues

• Closes [AWS] Filebeat s3 input polling mode does not backup to an S3 bucket #41070

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @kaiyan-sheng? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit

[mergify]

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b s3_input_backup upstream/s3_input_backup
git merge upstream/main
git push upstream s3_input_backup

[lucabelluccini]

Is there a test which would prevent this regression to occur again?