feat: expose InitChain tx responses #1941

n0izn0iz · 2024-04-17T13:54:32Z

Save and allow to fetch genesis txs responses

Before

❯ curl 'http://127.0.0.1:26657/block_results?height=0' -s | head -n 20
{
  "jsonrpc": "2.0",
  "id": "",
  "error": {
    "code": -32603,
    "message": "Internal error",
    "data": "height must be greater than 0"
  }
}

After

❯ curl 'http://127.0.0.1:26657/block_results?height=0' -s | head -n 20
{
  "jsonrpc": "2.0",
  "id": "",
  "result": {
    "height": "0",
    "results": {
      "deliver_tx": [
        {
          "ResponseBase": {
            "Error": null,
            "Data": null,
            "Events": null,
            "Log": "msg:0,success:true,log:,events:[]",
            "Info": ""
          },
          "GasWanted": "50000",
          "GasUsed": "240261825"
        },
        {
          "ResponseBase": {

Contributors' checklist...

Added new tests, or not needed, or not feasible
Provided an example (e.g. screenshot) to aid review or the PR is self-explanatory
Updated the official documentation or not needed
No breaking changes were made, or a BREAKING CHANGE: xxx message was included in the description
Added references to related issues and PRs
Provided any useful hints for running manual tests
Added new benchmarks to generated graphs, if any. More info here.

Signed-off-by: Norman Meier <norman@samourai.coop>

codecov · 2024-04-17T14:06:39Z

Codecov Report

Attention: Patch coverage is 54.54545% with 10 lines in your changes missing coverage. Please review.

Project coverage is 55.01%. Comparing base (4ff6bc7) to head (83dd5a0).

Files	Patch %	Lines
gno.land/pkg/gnoland/app.go	0.00%	8 Missing ⚠️
tm2/pkg/bft/rpc/core/blocks.go	80.00%	1 Missing ⚠️
tm2/pkg/bft/state/store.go	80.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #1941   +/-   ##
=======================================
  Coverage   55.01%   55.01%           
=======================================
  Files         595      595           
  Lines       79731    79744   +13     
=======================================
+ Hits        43861    43871   +10     
- Misses      32552    32555    +3     
  Partials     3318     3318

Flag	Coverage Δ
contribs/gnodev	`25.65% <ø> (-0.35%)`	⬇️
contribs/gnofaucet	`15.31% <ø> (+0.85%)`	⬆️
contribs/gnokeykc	`0.00% <ø> (ø)`
contribs/gnomd	`0.00% <ø> (ø)`
gno.land	`64.14% <0.00%> (-0.10%)`	⬇️
tm2	`54.43% <85.71%> (-0.04%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

gno.land/pkg/gnoland/app.go

moul · 2024-04-21T12:12:12Z

Is this common on other blockchains? I have concerns about the size of the response generated. I see advantages in having this new information, but I also think that it could be OK to just consider that "the genesis is there, and we don't need to know how."

n0izn0iz · 2024-04-21T19:46:15Z

I don't know if it's common in other blockchains but in order to index the portal loop we need to get the genesis transactions results otherwise indexers can't restore their state after a portal loop upgrade since the previous version transactions are embedded in genesis

moul · 2024-04-22T05:24:37Z

The Portal Loop isn't ideal for indexers. Indexers should focus on networks like test3 and test4. If we can enhance the Portal Loop to benefit other uses, we're open to it. Share how improving it could help beyond just the Portal Loop.

Related with #1791.

leohhhn · 2024-04-22T07:08:24Z

Hey @n0izn0iz, the Portal Loop is a bit specific when it comes to the way it works. It is not fully persistent will the way it handles state, and it can prove difficult to try and index it.

We discussed this with Berty as well - we can either 1. add a way to fetch genesis transactions, 2. think of a different solution to index the Portal Loop.

I'll bring this up on the engineer meeting today.

n0izn0iz · 2024-04-22T10:52:39Z

I also think it's important for other networks

What happens if you add a package in genesis that creates a grc20 token and want to index a "grc20.created" event, or "users.registered" events or dao membership history (some daos will probably be seeded in genesis, no?)

IMO there is many cases where it's desired to not have to extract this information in two ways (one for post-genesis-state via state inspection and one for regular txs via events) but instead run the same logic for all txs

#all_txs_are_equal 🙅🏽‍♂️

Maybe adding a gnoland subcommand to generate genesis result could be a way?
At least then the indexers could fetch the genesis and generate the results (but they would also need to know the correct chain binary for each chain)

zivkovicmilos

I'm a bit late to the party 🙏

I like this approach, and I think it works fine, but I think we can do even better -- please check out my leftover comments regarding data saves

tm2/pkg/bft/consensus/replay.go

Signed-off-by: Norman Meier <norman@samourai.coop>

…is-response

Signed-off-by: Norman Meier <norman@samourai.coop>

n0izn0iz · 2024-06-10T20:14:47Z

sorry for the push --force, I push-forced again to restore history before squash

Signed-off-by: Norman Meier <norman@samourai.coop>

tm2/pkg/bft/state/store.go

moul · 2024-06-19T17:32:37Z

What about a server starting with a quick sync snapshot, beginning at block 1M?

moul · 2024-06-19T17:33:58Z

The Genesis file is approximately 700MB on test3.

n0izn0iz · 2024-06-19T21:01:57Z

What about a server starting with a quick sync snapshot, beginning at block 1M?

I don't understand the question

The Genesis file is approximately 700MB on test3.

The full genesis goes in the ABCI protocol and it can be fetched from the api currently. I don't understand why we couldn't return and expose the results also

moul · 2024-06-24T12:05:27Z

What about a server starting with a quick sync snapshot, beginning at block 1M?
I don't understand the question

What happens when someone starts at block 1M instead of block 0? Will your virtual block 1M differ from other nodes that started from an earlier genesis? Could you add a test to expose this case for a chain starting with a sync snapshot?

The Genesis file is approximately 700MB on test3.
The full genesis goes in the ABCI protocol and it can be fetched from the api currently. I don't understand why we couldn't return and expose the results also

I think I don't fully understand the purpose of this PR. Could someone please explain the different elements (genesis, virtual block 0, and upcoming blocks) and how they are affected by this PR, perhaps with a visual schema?

tm2/pkg/bft/consensus/replay_test.go

tm2/pkg/bft/rpc/core/blocks.go

Signed-off-by: Norman Meier <norman@samourai.coop>

n0izn0iz · 2024-07-17T12:54:37Z

@moul

What happens when someone starts at block 1M instead of block 0? Will your virtual block 1M differ from other nodes that started from an earlier genesis? Could you add a test to expose this case for a chain starting with a sync snapshot?

How could I start at block 1M? I don't see a StartHeight field in any init types (neither GenesisDoc, nor InitChainRequest)

I think I don't fully understand the purpose of this PR. Could someone please explain the different elements (genesis, virtual block 0, and upcoming blocks) and how they are affected by this PR, perhaps with a visual schema?

The way a tendermint chain is initialized is through the InitChain ABCI endpoint

The in-memory GenesisDoc is passed to this call, in the case of gno (and most tendermint chains), this GenesisDoc contains transactions that are executed during the InitChain call

Currently, the result of these transactions is lost, this is very annoying for indexing as, among other things, you loose events from those transactions.

This PR returns the result of these genesis transactions in the InitChain response.

These results are stored as belonging to block #0 because the transactions in block #1 and subsequent blocks come from the mempool and are executed after the InitChain routine

tm2/pkg/bft/rpc/core/blocks.go

tm2/pkg/bft/abci/types/testing/mock_application.go

tm2/pkg/bft/state/store.go

Signed-off-by: Norman Meier <norman@samourai.coop>

Co-authored-by: Morgan <git@howl.moe>

n0izn0iz · 2024-07-23T11:42:44Z

❤️

Save and allow to fetch genesis txs responses ### Before ``` ❯ curl 'http://127.0.0.1:26657/block_results?height=0' -s | head -n 20 { "jsonrpc": "2.0", "id": "", "error": { "code": -32603, "message": "Internal error", "data": "height must be greater than 0" } } ``` ### After ``` ❯ curl 'http://127.0.0.1:26657/block_results?height=0' -s | head -n 20 { "jsonrpc": "2.0", "id": "", "result": { "height": "0", "results": { "deliver_tx": [ { "ResponseBase": { "Error": null, "Data": null, "Events": null, "Log": "msg:0,success:true,log:,events:[]", "Info": "" }, "GasWanted": "50000", "GasUsed": "240261825" }, { "ResponseBase": { ``` <details><summary>Contributors' checklist...</summary> - [ ] Added new tests, or not needed, or not feasible - [ ] Provided an example (e.g. screenshot) to aid review or the PR is self-explanatory - [ ] Updated the official documentation or not needed - [ ] No breaking changes were made, or a `BREAKING CHANGE: xxx` message was included in the description - [ ] Added references to related issues and PRs - [ ] Provided any useful hints for running manual tests - [ ] Added new benchmarks to [generated graphs](https://gnoland.github.io/benchmarks), if any. More info [here](https://github.com/gnolang/gno/blob/master/.benchmarks/README.md). </details> --------- Signed-off-by: Norman Meier <norman@samourai.coop> Co-authored-by: Morgan <git@howl.moe>

github-actions bot assigned n0izn0iz Apr 17, 2024

github-actions bot added 📦 🌐 tendermint v2 Issues or PRs tm2 related 📦 ⛰️ gno.land Issues or PRs gno.land package related labels Apr 17, 2024

n0izn0iz force-pushed the expose-genesis-response branch from b190ec1 to 9ed4da1 Compare April 17, 2024 13:56

feat: expose InitChain response

7be8225

Signed-off-by: Norman Meier <norman@samourai.coop>

n0izn0iz force-pushed the expose-genesis-response branch from 9ed4da1 to 7be8225 Compare April 17, 2024 13:58

Merge branch 'master' into expose-genesis-response

667f8bd

moul reviewed Apr 21, 2024

View reviewed changes

gno.land/pkg/gnoland/app.go Outdated Show resolved Hide resolved

moul mentioned this pull request Apr 22, 2024

Document various testnets and their goals #1791

Closed

Kouteki mentioned this pull request Apr 23, 2024

Minutes: Core Staff Weekly Syncs [every Monday] gnolang/meetings#36

Open

michelleellen mentioned this pull request May 6, 2024

Minutes: Public Dev Calls [every Wednesday] gnolang/meetings#37

Open

zivkovicmilos added the breaking change Functionality that contains breaking changes label May 29, 2024

zivkovicmilos requested changes May 29, 2024

View reviewed changes

tm2/pkg/bft/consensus/replay.go Outdated Show resolved Hide resolved

tm2/pkg/bft/consensus/replay.go Outdated Show resolved Hide resolved

jefft0 mentioned this pull request Jun 3, 2024

feat: index genesis transactions gnolang/tx-indexer#34

Merged

n0izn0iz added 4 commits June 10, 2024 21:07

tmp: start try to store results like in ApplyBlock

9107f45

Signed-off-by: Norman Meier <norman@samourai.coop>

chore: Merge remote-tracking branch 'origin/master' into expose-genes…

8bcee48

…is-response

feat: save InitChain's ResponseDeliverTxs and add test

8fa79be

Signed-off-by: Norman Meier <norman@samourai.coop>

chore: clean

e4dd0f6

Signed-off-by: Norman Meier <norman@samourai.coop>

n0izn0iz force-pushed the expose-genesis-response branch 2 times, most recently from 97733cb to e4dd0f6 Compare June 10, 2024 20:14

n0izn0iz changed the title ~~feat: expose InitChain response~~ feat: expose InitChain tx responses Jun 10, 2024

fix: preserve existing getHeight behavior

d0da512

Signed-off-by: Norman Meier <norman@samourai.coop>

n0izn0iz marked this pull request as ready for review June 10, 2024 21:10

n0izn0iz added 2 commits June 19, 2024 11:55

Merge branch 'master' into expose-genesis-response

9d309b7

Merge branch 'master' into expose-genesis-response

eda735b

moul reviewed Jun 19, 2024

View reviewed changes

tm2/pkg/bft/state/store.go Outdated Show resolved Hide resolved

Merge branch 'master' into expose-genesis-response

c5357f1

Kouteki requested a review from ajnavarro June 27, 2024 16:27

ajnavarro requested changes Jun 28, 2024

View reviewed changes

n0izn0iz and others added 4 commits July 17, 2024 14:03

Merge branch 'master' into expose-genesis-response

d684021

chore: use t.Cleanup

87fe7d8

Signed-off-by: Norman Meier <norman@samourai.coop>

chore: use cast instead of generic

8b33ffc

Signed-off-by: Norman Meier <norman@samourai.coop>

chore: add getHeight test

bad682a

Signed-off-by: Norman Meier <norman@samourai.coop>

ajnavarro approved these changes Jul 17, 2024

View reviewed changes

n0izn0iz added 4 commits July 17, 2024 21:56

Merge branch 'master' into expose-genesis-response

409cd9d

Merge branch 'master' into expose-genesis-response

5cec713

Merge branch 'master' into expose-genesis-response

ecdf0f2

Merge branch 'master' into expose-genesis-response

973a73f

thehowl approved these changes Jul 22, 2024

View reviewed changes

tm2/pkg/bft/rpc/core/blocks.go Outdated Show resolved Hide resolved

tm2/pkg/bft/abci/types/testing/mock_application.go Outdated Show resolved Hide resolved

tm2/pkg/bft/state/store.go Show resolved Hide resolved

n0izn0iz and others added 4 commits July 22, 2024 23:18

chore: revert getHeight

110da7d

Signed-off-by: Norman Meier <norman@samourai.coop>

chore: remove MockApplication

01dc6f8

Signed-off-by: Norman Meier <norman@samourai.coop>

chore: SaveABCIResponses internal usage note

e649de5

Co-authored-by: Morgan <git@howl.moe>

Merge branch 'master' into expose-genesis-response

83dd5a0

thehowl approved these changes Jul 23, 2024

View reviewed changes

thehowl merged commit 3aa0f0b into gnolang:master Jul 23, 2024
82 of 83 checks passed

n0izn0iz deleted the expose-genesis-response branch July 23, 2024 11:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: expose InitChain tx responses #1941

feat: expose InitChain tx responses #1941

n0izn0iz commented Apr 17, 2024 •

edited

Loading

codecov bot commented Apr 17, 2024 •

edited

Loading

moul commented Apr 21, 2024

n0izn0iz commented Apr 21, 2024 •

edited

Loading

moul commented Apr 22, 2024 •

edited

Loading

leohhhn commented Apr 22, 2024

n0izn0iz commented Apr 22, 2024 •

edited

Loading

zivkovicmilos left a comment

n0izn0iz commented Jun 10, 2024

moul commented Jun 19, 2024

moul commented Jun 19, 2024

n0izn0iz commented Jun 19, 2024 •

edited

Loading

moul commented Jun 24, 2024

n0izn0iz commented Jul 17, 2024 •

edited

Loading

n0izn0iz commented Jul 23, 2024

feat: expose InitChain tx responses #1941

feat: expose InitChain tx responses #1941

Conversation

n0izn0iz commented Apr 17, 2024 • edited Loading

Before

After

codecov bot commented Apr 17, 2024 • edited Loading

Codecov Report

moul commented Apr 21, 2024

n0izn0iz commented Apr 21, 2024 • edited Loading

moul commented Apr 22, 2024 • edited Loading

leohhhn commented Apr 22, 2024

n0izn0iz commented Apr 22, 2024 • edited Loading

zivkovicmilos left a comment

Choose a reason for hiding this comment

n0izn0iz commented Jun 10, 2024

moul commented Jun 19, 2024

moul commented Jun 19, 2024

n0izn0iz commented Jun 19, 2024 • edited Loading

moul commented Jun 24, 2024

n0izn0iz commented Jul 17, 2024 • edited Loading

n0izn0iz commented Jul 23, 2024

n0izn0iz commented Apr 17, 2024 •

edited

Loading

codecov bot commented Apr 17, 2024 •

edited

Loading

n0izn0iz commented Apr 21, 2024 •

edited

Loading

moul commented Apr 22, 2024 •

edited

Loading

n0izn0iz commented Apr 22, 2024 •

edited

Loading

n0izn0iz commented Jun 19, 2024 •

edited

Loading

n0izn0iz commented Jul 17, 2024 •

edited

Loading