Refactor Publishing #34
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
cc @kyle-dorman I can't add you as a reviewer, not sure why |
Lints were previously set to run for each platform, which isn't particularly useful. They're now on ubuntu only Additionally the lints were run without features enabled, which would miss most of the code. Now they run with all features. We could consider running a lint matrix against some subset of features too, but I'm not sure if that would be useful. This exposed a few places in the code which failed the now-feature-checking lints, these have been fixed
This changeset completely refactors publishing to be based on async Sinks, and to port the googlepubsub publisher to use gRPC instead of the JSON REST api. Together, these should make the library easier to use, compose better with the broader rust async ecosystem, and provide better performance for publishing. This additionally simplifies the crate hierarchy and some of the crate features, as the divisions only made much sense when publishing and consuming were separate implementations. Now backends are expected to provide publishing and consuming together at the same time. Most of this code is adapted from the private crate standard-hedwig, which has been a proving ground for the publishing API before it could be incorporated into this open-source crate, and has seen effective operation in production. There are still some changes to follow in later commits to finalize the API before publishing as a major release, as noted in some of the source comments. The general shape of the library is largely ready, however, and is worth beginning a review.
This resolves some lint name problems and provides BTreeMap::retain
This change adds mechanisms to 1) inform users when a (possibly buffered) message has been published, and 2) retry operations, both now in the user's message type. Previously retries were only possible in an opaque pubsub grpc type, and the publish listening wasn't available.
nagisa
reviewed
Oct 27, 2021
examples/googlepubsub.rs
Outdated
| } | ||
|
|
||
| #[derive(Debug)] | ||
| struct TransformedMessage { |
There was a problem hiding this comment.
This seems like something that might make sense to have provided by the library in a generic way, right?
There was a problem hiding this comment.
There's the AcknowledgeableMessage type that the user could employ (and a pubsub-specific alias PubSubMessage). Now that i think about it, I should alias the TransformedMessage type to one of those.
I've also thought about adding a map function to AcknowledgeableMessage so that a user could operate on the message and leave the token as-is, but then this goes down the rabbit hole of monad-ish types (implement flatmap? flatten? fallible mapping? async mapping?). Instead i left the fields as public so the user can construct them as they wish
- print error descriptions in PublishError::Display - use PubSubMessage type for carrying ack tokens in pubsub example
nagisa
approved these changes
Oct 27, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This changeset completely refactors publishing to be based on async
Sinks, and to port the googlepubsub publisher to use gRPC instead of the
JSON REST api. Together, these should make the library easier to use,
compose better with the broader rust async ecosystem, and provide better
performance for publishing.
This additionally simplifies the crate hierarchy and some of the crate
features, as the divisions only made much sense when publishing and
consuming were separate implementations. Now backends are expected to
provide publishing and consuming together at the same time.
Most of this code is adapted from the private crate standard-hedwig,
which has been a proving ground for the publishing API before it could
be incorporated into this open-source crate, and has seen effective
operation in production.
There are still some changes to follow in later commits to finalize theChanges added, this is all set to go.API before publishing as a major release, as noted in some of the source
comments. The general shape of the library is largely ready, however,
and is worth beginning a review.