CassandraSinkCluster: Keep node list up to date via a tokio::watcher #831
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #730
This PR ensures that all existing connections receive all changes to the topology tasks node list.
Previously a new connection had to be made to observe the changes to the node list.
It also simplies code in a few places that had to rely on loop+sleep to wait for changes to occur, now we can just await on the recv call.
renames:
nodes_rx
also pairs nicely to thenodes
field to which it is copied.local
part no longer makes sense now that we are using a Receiver, additionally local could be confused with thelocal_shotover_node
field.Alternative approach
I originally attempted to implement this with an enum that gets sent from the topology task to the transforms providing granular updates:
It allows us to avoid copying an entire node list in many cases.
However these cases are error cases and our performance doesnt have to be amazing there.
And it comes at the cost of a large increase to complexity in both the topology task side and the transform side.
While I believe we could write a correct implementation with this approach without too much trouble, it is still harder to reason about, harder to quickly verify correctness and harder to maintain.
Additionally the main cost would be having to recreate connections, and I've added logic to this PR to copy over the connections from the old list.
So in the end I decided it wasnt worth it and went with the much simpler approach in this PR.