[tendermint2] re-implement tx indexing #275
Assignees
Labels
Milestone
Comments
|
copied here: tendermint/tendermint2#2 |
11 tasks
ajnavarro
added
🌱 feature
18 tasks
zivkovicmilos
added a commit
that referenced
this issue
Oct 19, 2023
# Description This PR introduces file-based transaction indexing. As discussed in #275, transaction index parsing should be done as a separate process from the main node, meaning other services can be instantiated to index transactions as readers. The general architecture of the transaction indexers in this PR can be described with the following image: <img width="1481" alt="Architecture" src="https://user-images.githubusercontent.com/16712663/221845299-ff552470-8efc-4134-8c3d-e71e74929acc.png"> Each concrete indexer implementation decides how to handle transaction events, and where to store them. Independent processes from the indexers themselves read these events (by parsing files, logs, executing RPC queries...). ## File Indexer The `file` transaction indexer that is included in this PR utilizes `autofile.Group`s to write down transaction events. Users can now specify to use the file-based indexer with the following added flags to the `gnoland` command: - `--tx-indexer-type` - specify the type of indexer (none is default) - `--tx-indexer-path` - path for the file-based tx indexer # Changes include - [ ] Bugfix (non-breaking change that solves an issue) - [ ] Hotfix (change that solves an urgent issue, and requires immediate attention) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (change that is not backwards-compatible and/or changes current functionality) # Checklist (for contributors) - [x] I have assigned this PR to myself - [x] I have added at least 1 reviewer - [x] I have added the relevant labels - [ ] I have updated the official documentation - [x] I have added sufficient documentation in code # Testing - [x] I have tested this code with the official test suite - [x] I have tested this code manually ## Manual tests - Manually executed transactions and verified they were saved to disk. - Added unit tests that cover all added functionality. # Additional comments - [Relevant tendermint2 issue](tendermint/tendermint2#2) - Resolves #275 EDIT: After comments from @jaekwon, this `Indexer` functionality has been renamed to `EventStore`, and work on an independent indexer process (process that can read from the event store) will begin soon that will offer indexing functionality cc @ilgooz
thehowl
pushed a commit
to thehowl/gno
that referenced
this issue
Oct 21, 2023
# Description This PR introduces file-based transaction indexing. As discussed in gnolang#275, transaction index parsing should be done as a separate process from the main node, meaning other services can be instantiated to index transactions as readers. The general architecture of the transaction indexers in this PR can be described with the following image: <img width="1481" alt="Architecture" src="https://user-images.githubusercontent.com/16712663/221845299-ff552470-8efc-4134-8c3d-e71e74929acc.png"> Each concrete indexer implementation decides how to handle transaction events, and where to store them. Independent processes from the indexers themselves read these events (by parsing files, logs, executing RPC queries...). ## File Indexer The `file` transaction indexer that is included in this PR utilizes `autofile.Group`s to write down transaction events. Users can now specify to use the file-based indexer with the following added flags to the `gnoland` command: - `--tx-indexer-type` - specify the type of indexer (none is default) - `--tx-indexer-path` - path for the file-based tx indexer # Changes include - [ ] Bugfix (non-breaking change that solves an issue) - [ ] Hotfix (change that solves an urgent issue, and requires immediate attention) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (change that is not backwards-compatible and/or changes current functionality) # Checklist (for contributors) - [x] I have assigned this PR to myself - [x] I have added at least 1 reviewer - [x] I have added the relevant labels - [ ] I have updated the official documentation - [x] I have added sufficient documentation in code # Testing - [x] I have tested this code with the official test suite - [x] I have tested this code manually ## Manual tests - Manually executed transactions and verified they were saved to disk. - Added unit tests that cover all added functionality. # Additional comments - [Relevant tendermint2 issue](tendermint/tendermint2#2) - Resolves gnolang#275 EDIT: After comments from @jaekwon, this `Indexer` functionality has been renamed to `EventStore`, and work on an independent indexer process (process that can read from the event store) will begin soon that will offer indexing functionality cc @ilgooz
gfanton
pushed a commit
to gfanton/gno
that referenced
this issue
Nov 9, 2023
# Description This PR introduces file-based transaction indexing. As discussed in gnolang#275, transaction index parsing should be done as a separate process from the main node, meaning other services can be instantiated to index transactions as readers. The general architecture of the transaction indexers in this PR can be described with the following image: <img width="1481" alt="Architecture" src="https://user-images.githubusercontent.com/16712663/221845299-ff552470-8efc-4134-8c3d-e71e74929acc.png"> Each concrete indexer implementation decides how to handle transaction events, and where to store them. Independent processes from the indexers themselves read these events (by parsing files, logs, executing RPC queries...). ## File Indexer The `file` transaction indexer that is included in this PR utilizes `autofile.Group`s to write down transaction events. Users can now specify to use the file-based indexer with the following added flags to the `gnoland` command: - `--tx-indexer-type` - specify the type of indexer (none is default) - `--tx-indexer-path` - path for the file-based tx indexer # Changes include - [ ] Bugfix (non-breaking change that solves an issue) - [ ] Hotfix (change that solves an urgent issue, and requires immediate attention) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (change that is not backwards-compatible and/or changes current functionality) # Checklist (for contributors) - [x] I have assigned this PR to myself - [x] I have added at least 1 reviewer - [x] I have added the relevant labels - [ ] I have updated the official documentation - [x] I have added sufficient documentation in code # Testing - [x] I have tested this code with the official test suite - [x] I have tested this code manually ## Manual tests - Manually executed transactions and verified they were saved to disk. - Added unit tests that cover all added functionality. # Additional comments - [Relevant tendermint2 issue](tendermint/tendermint2#2) - Resolves gnolang#275 EDIT: After comments from @jaekwon, this `Indexer` functionality has been renamed to `EventStore`, and work on an independent indexer process (process that can read from the event store) will begin soon that will offer indexing functionality cc @ilgooz
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The tx indexing and query system was removed to attempt a more modular and minimal implementation.
The goal I have in mind is to simply log to file all EventSwitch events (serialized using amino:binary, using protobuf for speed), and then to make readers that can idempotent-ly index or filter these events for whatever purpose, including tx indexing, but also streaming of filtered events (from the RPC using websockets, say). These streaming readers would be independent processes from the full node, so there would be possibly no resource contention depending on the configuration. Since log files could be read by any process, the node wouldn't even have to know who are the listeners are.
We could possibly use an existing persistent pubsub system to implement this, but if we're going to use such an external dependency, it needs to be implemented in Go, be minimal and simple, and mature. Alternatively we should implement one from scratch if a suitable implementation cannot be found.
The log file could use pkgs/autofile, or a fork that doesn't automatically delete older chunk files, and we could leave it up to the user to run a background service that deletes older chunk files as needed.
I'm open to alternative solutions, but this seems to be the most conceptually simple solution. Compared to Tendermint1, instead of an asynchronous pubsub, Tendermint2 has the original synchronous simple event switch. Tendermint1 bakes in several pubsub listeners, but Tendermint2 would be more minimal and simply log all events to the filesystem.
The text was updated successfully, but these errors were encountered: