feat(geyser): support dumping accounts to csv #279
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* channel: replace channel with a better implementation * start integrating channel * add some test cases for the channel * only look at the final `gossip_table` when the we've exited * fix even more races in the test suite * enable tsan in the test CI * remove all usages of `unordered` from the codebase * removing PacketBatch in favour of sending singular packets * choose better time units for benchmarks * don't use identifiers that start with a underscore * remove the old channel implementation * add some documentation and clean up the code * remove stale code and improve gossip value filtering * implement an exit order sync method * cleanup allocation even more * remove the last usages of close * clarify `CONTRIBUTING.md` and more `initial_capacity` from the channel initialization * gossip: revert filter optimization - Also don't take ownership of the pull request gossip values, they aren't ours to free. This was causing GPA double free detection to trigger * gossip: add back missing `clearRetainingCapacity` * grafana: fix overlapping timestamps * fix racy deinit for gossip service * add some comments * remove `.discard` flag and fix up some logic * remove unused variable * remove UAF from gossip So this one was really hard to find. In Zig, storing the pointer (and then using it) to a capture which was by-value is not well defined. LLVM is allowed to create induction stack space for optimizing loops and overwrite the stack space where the pointer was pointing at. Here we are storing a pointer to the capture, probably as an premature optimization, and it was getting overwriten. Storing a pointer to the prune message wouldn't have optimized anyways, since the capture is still a stack-copy. * remove unneeded free * fix shred collector exiting early * cmd: remember to join on errors so they don't keep running * fix(shred-collector): not processing shreds there was a bool being used to decide whether to pass packets to the shred verifier, and it was being negated too many times. i fixed the logic. i think the bug was introduced due to having the logic spread across too many scopes, which made it confusing. i tried to make it clearer by moving the logic into a narrower scope --------- Co-authored-by: Drew Nutter <dnut@users.noreply.github.com>
0xNineteen
force-pushed
the
19/snapshot-dump
branch
from
66869b1
to
038a391
Compare
InKryption
requested changes
Oct 2, 2024
Rexicon226
requested changes
Oct 2, 2024
InKryption
previously approved these changes
Oct 2, 2024
|
I'd like a chance to take a glance at the Grafana, but otherwise the code is good for me. |
|
I played around with the Grafana a bit, and I don't like the ugly overlapping values. It's also really unclear what those values mean. What I personally think makes the most sense is to
this is what my end result looks like: |
|
nice - changed to use rate + bar and added docs explaining the metrics (see new description) probs want to document every metric we are recording 🤔 |
InKryption
approved these changes
Oct 3, 2024
0xNineteen
added a commit
that referenced
this pull request
Oct 9, 2024
dump a snapshot loading to csv using geyser -- main command: ./zig-out/bin/geyser csv * new tryCollapse method on recycle fba to collapse adjacent free records * add docs on new usage * add cli setup for geyser reader (src/geyser/bin.zig) * add new EndOfSnapshotLoading enum to payload type * add new geyser metrics to grafana
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
dump a snapshot loading to csv using geyser
main command:
./zig-out/bin/geyser csvnote: full dump of testnet snapshot is 75GB
tryCollapsemethod on recycle fba to collapse adjacent free recordssrc/geyser/bin.zig)