This is the beginning of the clustering stuff. Client/server communication, etc. #20
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Feedback... "yay". 🍒 |
|
/cc @phobos182 @dysinger because they know a lot more about Go than I do. |
ghost
assigned 
… client and server for cluster communication. Fill out request/response in protocol. Add request handler interface and implementation.
…ything together. Move the server daemon to damon/infuxd.go. Remove extraneous stuff from the client server test.
…object from http/api over to cluster config since it makes more sense there.
…quest handler. Add LogRequest method stub to datastore.
Conflicts: src/datastore/leveldb_datastore.go
| required int64 timestamp = 2; | ||
| required uint32 sequence_number = 3; | ||
| optional int64 timestamp = 2; | ||
| optional uint32 sequence_number = 3; | ||
| } |
There was a problem hiding this comment.
just a reminder. per our conversation, these will change back to required fields and timestamp/sequence number will be assigned by the server that received the initial request before proxying (possibly locally) the points to other nodes.
|
💖 |
| if ok { | ||
| responseChan <- response | ||
| if *response.Type == protocol.Response_END_STREAM { | ||
| close(responseChan) |
There was a problem hiding this comment.
Should the channel be deleted from the map as well ?
Actually related issue, what happens to requests that are lost for one reason or another. We need some sort of garbage collector to go back and clean up the map after a certain timeout has elapsed.
There was a problem hiding this comment.
yep, it should be. Will probably update this to use gocache.
* Move ReplicateWrite to live only on Coordinator. * Change SequenceNumber on Point to be a uint64 to support higher numbers and host ids being the first part of it. * Move time and sequence_number assigning to the coordinator instead of in the datastore. * Roll RequestLogging and PersistentAtomicIncrement into the Datastore interface. * Add PersistentAtomicIncrement to LevelDbDatastore to support non-colliding sequence numbers.
…sts when responses are sent.
| } | ||
|
|
||
| func (self *ProtobufClient) reconnect() { | ||
| swapped := atomic.CompareAndSwapUint32(&self.reconnecting, self.reconnecting, IS_RECONNECTING) |
There was a problem hiding this comment.
I think the second argument should be IS_CONNECTED, otherwise the compare will always be true.
| if err != nil { | ||
| // we failed to write locally and to any proxy, bail | ||
| return err | ||
| } |
There was a problem hiding this comment.
what happens if the local write failed and the proxy write succeed ? are we going to send the replicas another write ?
There was a problem hiding this comment.
The one that took the proxy will be responsible for replicating to the other nodes. If those fail, the downed node will replay missing data when it comes back online.
There was a problem hiding this comment.
ah, see what you mean, fixing.
the integration tests and the package script should be using daemon instead of server which is the old name of the influxdb binary
* Update Response in protocol to have nextPointTime * Wire up distribute query in coordinator * Add method to ClusterCongig to get servers to query * Add query handling to request handler * Add point sorting to protocol.Series
Conflicts: src/datastore/leveldb_datastore.go src/engine/engine.go
…striubted query bugs. 3 days of debugging pays off in glorious success! The ring filter for datastore is optional to have it filter what points get yielded by their location in the ring. This needs to be used when making distributed queries on a database whose replication factor doesn't evenly divide the size of the cluster.
…owner server id, and writing server id for use in request logging and replication.
…der than 48 hours.
Conflicts: package.sh
* Update protocol with replication log replays. * Update protobuf server to keep map of connections to be able to close cleanly * Add replay support to request handler, coordinator, and datastore
jvshahid
pushed a commit
that referenced
this pull request
Aug 12, 2014
mgattozzi
added a commit
that referenced
this pull request
Sep 5, 2024
* fix: wal skip persist and notify if empty buffer (#25211) * fix: wal skip persist and notify if empty buffer This fixes the WAL so that it will skip persisting a file and notifying the file notifier if the wal buffer is empty. * fix: fix last cache persist test * fix: make ParquetChunk fields and mod chunk pub (#25219) * fix: make ParquetChunk fields and mod chunk pub This doesn't affect anything in the OSS version, but these changes are needed for Pro as part of our compactor work. * fix: cargo deny failure * fix: query bugs with buffer (#25213) * fix: query bugs with buffer This fixes three different bugs with the buffer. First was that aggregations would fail because projection was pushed down to the in-buffer data that de-duplication needs to be called on. The test in influxdb3/tests/server/query.rs catches that. I also added a test in write_buffer/mod.rs to ensure that data is correctly queryable when combining with different states: only data in buffer, only data in parquet files, and data across both. This showed two bugs, one where the parquet data was being doubled up (parquet chunks were being created in write buffer mod and in queryable buffer. The second was that the timestamp min max on table buffer would panic if the buffer was empty. * refactor: PR feedback * fix: fix wal replay and buffer snapshot Fixes two problems uncovered by adding to the write_buffer/mod.rs test. Ensures we can replay wal data and that snapshots work properly with replayed data. * fix: run cargo update to fix audit * feat: use host identifier prefix in object store paths (#25224) This enforces the use of a host identifier prefix in all object store paths (currently, for parquet files, catalog files, and snapshot files). The persister retains the host identifier prefix, and uses it when constructing paths. The WalObjectStore also holds the host identifier prefix, so that it can use it when saving and loading WAL files. The influxdb3 binary requires a new argument 'host-id' to be passed that is used to specify the prefix. * feat: add `system.parquet_files` table (#25225) This extends the system tables available with a new `parquet_files` table which will list the parquet files associated with a given table in a database. Queries to system.parquet_files must provide a table_name predicate to specify the table name of interest. The files are accessed through the QueryableBuffer. In addition, a test was added to check success and failure modes of the new system table query. Finally, the Persister trait had its associated error type removed. This was somewhat of a consequence of how I initially implemented this change, but I felt cleaned the code up a bit, so I kept it in the commit. * fix: un-pub QueryableBuffer and fix compile errors (#25230) * refactor: Make Level0Duration part of WAL (#25228) * refactor: Make Level0Duration part of WAL I noticed this during some testing and cleanup with other PRs. The WAL had its own level_0_duration and the write buffer had a different one, which would cause some weird problems if they weren't the same. This refactors Level0Duration to be in the WAL and fixes up the tests. As an added bonus, this surfaced a bug where multiple L0 blocks getting persisted in the same snapshot wasn't supported. So now snapshot details can have many files per table. * fix: have persisted files always return in descending data time order * fix: sort record batches for test verification * fix: main (#25231) * feat: Add last cache create/delete to WAL (#25233) * feat: Add last cache create/delete to WAL This moves the LastCacheDefinition into the WAL so that it can be serialized there. This ended up being a pretty large refactor to get the last cache creation to work through the WAL. I think I also stumbled on a bug where the last cache wasn't getting initialized from the catalog on reboot so that it wouldn't actually end up caching values. The refactored last cache persistence test in write_buffer/mod.rs surfaced this. Finally, I also had to update the WAL so that it would persist if there were only catalog updates and no writes. Fixes #25203 * fix: typos * feat: Catalog apply_catalog_batch only updates if new (#25236) * feat: Catalog apply_catalog_batch only updates if new This updates the Catalog so that when applying a catalog batch it only updates the inner catalog and bumps the sequence number and updated tracker if there are new updates in the batch. Also does validation that the catalog batch schema is compatible with any existing. Closes #25205 * feat: only persist catalog when updated (#25238) * chore: ignore sqlx rustsec advisory (#25252) * feat: Add FileIndex type to influxdb3_index This commit does two important things: 1. It creates a new influxdb3_index crate under influxdb3_pro to contain all indexing logic and types that we might create for influxdb3_pro 2. Creates our first index type the FileIndex which is part of #20 Note we're starting off with just file ids as this will let us set up the logic for creating and working with the `FileIndex` inside of the compactor first. Later we can add row groups as that logic is a bit more complicated in nature. The `FileIndex` contains methods to lookup, insert, and delete items from the index as needed and an associated test to make sure it works as expected. Note that the `FileIndex` is meant to have one created for each database table that has an index created for it. Later on when it's being integrated into the compactor a `FileIndex` will be returned per compaction of a given table. We'll later integrate this into the `WriteBuffer` for querying as well as adding this to the WAL so that indexes can be recreated as needed. --------- Co-authored-by: Paul Dix <paul@pauldix.net> Co-authored-by: Trevor Hilton <thilton@influxdata.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Don't bother merging this in yet. Just review and give feedback please :)