feat(Gossip KV): Merge Gossip KV into main.
#1535
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Included in this commit: * Elementary kubernetes YAMLs * Instructions in `README.md` to install, build images and deploy to minikube * Overall `README.md` (extremely rough draft) of the architecture of the system
… store. (#1246) The data model supported by the layer 0 key-value store is analogous to the following JSON: ```json { "sys": [ "members": [ "key": "value" ] ], "usr": [ "table_a": [ ... ], "table_b": [ ... ], ] } ``` A single `TableMap` can model `sys` or `usr`. We're only planning for these two namespaces for now, so no higher-level lattices allow for more namespaces. Initially, I started with support for deleting an entire table efficiently, but I have decided to drop that until needed. If it isn't needed in practice, the lack of a sharp operation that can delete an entire table leads to a safer system. It also simplified the code a fair bit. The tests may feel like they're testing lattice merging (and therefore redundant), but I wrote these to convince myself that the type signatures of the data model are working. I may remove them in the future (or not!).
It was not the most critical thing to get in, but it bothered me a bit. This code change allows '/' to be used in table and row names. This doesn't affect memory layout in any way. The most critical part of this change is the additional tests to check for empty table names and row keys.
…tiple values for concurrent writes) (#1275) These changes separate the server logic from the networking logic, allowing the server logic to be tested with high fidelity in a unit test. The changes allow for an abstract replacement of `SocketAddr`, which can be used for deterministic testing of more complex scenarios in unit tests. The first feature to use these testing capabilities is concurrent writes to the same key at the same tick, which should result in multiple values being returned from a `Get`.
* Added support for hierarchical configuration using `config` crate & used it to set seed node information * Some cleanup for types & interfaces
# Conflicts: # Cargo.lock
This change implements a blind gossip protocol where the gossipers don't receive (or care to receive) responses from the systems they gossip with. It also adds a deterministic testing framework for testing gossip. Once request-reply is added, it can stop being blind. Co-authored-by: Mingwei Samuel <mingwei.samuel@gmail.com>
# Conflicts: # Cargo.lock
This ensures that an infected node continues to infect other nodes until it loses "interest" (because the nodes it contacted are already infected).
# Conflicts: # hydroflow_lang/src/graph/ops/state.rs
When I started, I was using the published Hydroflow. I have since switched to using local hydroflow, which means the Dockerfile needs to be updated for the build to work correctly. This posed a more significant challenge than expected, primarily because of disk space issues, which have been sorted out.
Summary of Changes: 1. Some cleanup/bug fixes with docker files. 2. Install `dig` on CLI to test DNS 3. Fixed gossip ports everywhere - using the wrong port was annoying. 4. Cluster tear-down is much faster - reduced pod termination wait from 30s (default) to 5s. 5. New static config for three servers - not planning to automate this now. 6. Gossip trigger timer wired in - this was earlier separated out for deterministic testing. 7. Not randomizing host-names at the moment - this was making testing difficult, although I think this does help in production with restarts and such. 8. Gossip assumed > 0 available peers - fixed that issue.
Remove extra target which exists by two names.
I created a load-testing binary with an in-process network—deployed and tested with five threads @ one request per second per thread.
# Conflicts: # Cargo.lock
rohitkulshreshtha
added
the
Datastores/Gossip-KV
Deploying hydroflow with
|
| Latest commit: |
8a5890e
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://cdb26d63.hydroflow.pages.dev |
| Branch Preview URL: | https://1533-gossip-kv-remove-code-d.hydroflow.pages.dev |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There are some TODOs in the code for which I have created issues for 0.10 milestone.
P0 - Must Have for 0.10
P1 - Good to have / stretch goals
P2 - Low Priority
Closes #1530.