Change File info poller to respond when capacity in channel opens up and decode stream into Vec of structs #715
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bbalser
force-pushed
the
bbalser/file-info-poller-updates
branch
from
b5fde69 to
69d38a2
Compare
jeffgrunewald
approved these changes
Jan 22, 2024
| } | ||
| } | ||
| _ = cleanup_trigger.tick() => self.clean(&self.cache).await?, | ||
| result = futures::future::try_join(sender.reserve().map_err(Error::from), self.get_next_file()) => { |
There was a problem hiding this comment.
looks like it's safe to assume that the joined result tuple is treated like collecting an iterator of results into a single result wrapping a collection, i.e. assuming both operations being joined are Ok(_) the result tuple will be Ok(result_1_inner, result_2_inner)?
There was a problem hiding this comment.
That is my understanding
https://docs.rs/futures/0.3.30/futures/future/fn.try_join.html
maplant
approved these changes
Jan 22, 2024
andymck
approved these changes
Jan 23, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is an attempt to make the file_info_poller more responsive to capacity in the mpsc::channel. It should help consumers not have any delays in processing data when they are backed up.
Also the file_info_poller now converted the stream of bytes into a Vec of the decoded struct. Not sure this is a great idea, but we have seen that holding the stream open to AWS while in the channel waiting for the consumer to process the file can cause "connection reset by peer" errors when the consumer takes a while processing the stream or is really backed up. This change will cause the use of more memory but should allow the consumer to process data faster.