[AWS] improve S3 input states copy by only storing filtered entries

[Kavindu-Dodan]

Proposed commit message

Improves S3 polling mode state registry copy by considering the bucket_list_prefix usage. Before this change, input stored all registry entries loaded from the underlying registry (ex:- when storing, restarting beats or upgrading while pointing to the same registry). With this improvement, when bucket_list_prefix is used, the state registry copy only holds entries matching the given prefix.

This improvement benefits the state registry clean-up planned through #41694. Further, when beats restart, this change improves the input-specific state copy memory usage by only storing relevant entries, matching the configured prefix.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

None - this change only concerns the input-specific copy of the registry entries

How to test this PR locally

Requires a build and S3 bucket entries with prefixes.

• Generate a mix of S3 bucket entries with prefixes and without. You may use data gen tool ¹
• Build filebeat from this branch & configure multiple inputs with prefixes
• Observe state registry filling at startup. You may first run without a prefix to store all entries and later restart beats to observe the loading behavior

Related issues

#39116

Footnotes

1. https://github.com/Kavindu-Dodan/data-gen ↩

[elasticmachine]

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)

[kaiyan-sheng]

The change looks good. Just one question: if we have two s3 input running, one with prefix A and one without any prefix. Will the state be missing entries?

[Kavindu-Dodan]

The change looks good. Just one question: if we have two s3 input running, one with prefix A and one without any prefix. Will the state be missing entries?

No, there won't be missing entries. The prefixed input store will store and handle entries with prefixes. The non-prefixed input will maintain all entries, including prefixed ones. This holds even when restarting/upgrading, pointing to the same registry.