add listPrefix in awsS3WriteCommitPrefix #31776
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
botelastic
bot
added
the
needs_team
botelastic
bot
removed
the
needs_team
💚 Build Succeeded
Expand to view the summary
Build stats
Test stats 🧪
💚 Flaky test reportTests succeeded. 🤖 GitHub commentsTo re-run your PR in the CI, just comment with:
|
aspacca
commented
May 31, 2022
| continue | ||
| } | ||
|
|
||
| var latestStoredTime time.Time | ||
| keys[state.ID] = struct{}{} | ||
| latestStoredTime, ok := latestStoredTimeByBucket[state.Bucket] | ||
| latestStoredTime, ok := latestStoredTimeByBucketAndListPrefix[state.Bucket+state.ListPrefix] |
There was a problem hiding this comment.
this is indeed both a bugfix and a breaking change
from the elastic side the only integrations that does that is the one on cisco managed s3 buckets, that's indeed the bugged integration that will fix
but in the case of users using custom setup with a single bucket and multiple list prefixes it will be a breaking change as in the measure that s3 objects could be ingested again (and probably fails ingestion because of same document ids)
P1llus
approved these changes
May 31, 2022
aspacca
added
backport-v8.2.0
|
/test |
mergify bot
pushed a commit
that referenced
this pull request
Jun 2, 2022
* add listPrefix in awsS3WriteCommitPrefix * linting and changelog * linting * try fixing integration test * try fixing integration test (cherry picked from commit ee38609) # Conflicts: # x-pack/filebeat/input/awss3/input_benchmark_test.go
mergify bot
pushed a commit
that referenced
this pull request
Jun 2, 2022
* add listPrefix in awsS3WriteCommitPrefix * linting and changelog * linting * try fixing integration test * try fixing integration test (cherry picked from commit ee38609) # Conflicts: # x-pack/filebeat/input/awss3/input_benchmark_test.go
rdner
pushed a commit
that referenced
this pull request
Jun 2, 2022
* add listPrefix in awsS3WriteCommitPrefix (#31776) * add listPrefix in awsS3WriteCommitPrefix * linting and changelog * linting * try fixing integration test * try fixing integration test (cherry picked from commit ee38609) # Conflicts: # x-pack/filebeat/input/awss3/input_benchmark_test.go * fix backport * remove duplicates in test Co-authored-by: Andrea Spacca <andrea.spacca@elastic.co>
aspacca
pushed a commit
that referenced
this pull request
Jun 2, 2022
) * add listPrefix in awsS3WriteCommitPrefix (#31776) * add listPrefix in awsS3WriteCommitPrefix * linting and changelog * linting * try fixing integration test * try fixing integration test (cherry picked from commit ee38609) # Conflicts: # x-pack/filebeat/input/awss3/input_benchmark_test.go * fix backport * remove duplicates in test Co-authored-by: Andrea Spacca <andrea.spacca@elastic.co>
mergify bot
pushed a commit
that referenced
this pull request
Jun 21, 2022
* add listPrefix in awsS3WriteCommitPrefix * linting and changelog * linting * try fixing integration test * try fixing integration test (cherry picked from commit ee38609)
aspacca
pushed a commit
that referenced
this pull request
Jun 23, 2022
* add listPrefix in awsS3WriteCommitPrefix (#31776) * add listPrefix in awsS3WriteCommitPrefix * linting and changelog * linting * try fixing integration test * try fixing integration test (cherry picked from commit ee38609) * Update CHANGELOG.next.asciidoc * Update CHANGELOG.next.asciidoc Co-authored-by: Andrea Spacca <andrea.spacca@elastic.co>
chrisberkhout
pushed a commit
that referenced
this pull request
Jun 1, 2023
* add listPrefix in awsS3WriteCommitPrefix * linting and changelog * linting * try fixing integration test * try fixing integration test
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Bug
What does this PR do?
Fix
awsS3WriteCommitPrefixbug when using a single bucket with multiple listPrefix on aws-s3 s3 direct inputWhy is it important?
Regarding the Cisco Umbrella integration via the elastic agent, a customer ingesting using the Cisco Managed S3 and have near 40 individual integrations setup for different customers. However they found logs to be missing as though the agent failed to read them, not even partial parsing of a log. I checked the files within an S3 bucket to make sure they were written by Cisco and they were.
Checklist
- [ ] I have made corresponding changes to the documentation- [ ] I have made corresponding change to the default configuration filesCHANGELOG.next.asciidocorCHANGELOG-developer.next.asciidoc.Author's Checklist
How to test this PR locally
Related issues
Use cases
Screenshots
Logs