[AWS] Filebeat s3 input polling mode does not backup to an S3 bucket

[kaiyan-sheng]

When running Filebeat s3 input polling mode, we suppose to have the options to backup s3 objects into a specific bucket and delete the object after backup.

filebeat.inputs:
- type: aws-s3
  enabled: true
  credential_profile_name: elastic-observability
  bucket_arn: arn:aws:s3:::test-s3-bucket-ks
  default_region: eu-central-1
  backup_to_bucket_arn: arn:aws:s3:::test-s3-bucket-ks-backup
  delete_after_backup: true
  number_of_workers: 5

With this config, objects/logs will be ingested from the bucket specified in bucket_arn into Elasticsearch just fine but no backup or delete happened.

This is a bug introduced in 8.14

[elasticmachine]

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)