common: Parse post-merge hardfork config in parseGethParams and handle post-merge genesis block #2427

g11tech · 2022-11-17T10:47:48Z

In order to get the geth genesis working for 4844 work, the sharding block needs to be configured post merge which is configured post block 0 (Currently merge if present is the last hf configured)

Similar setup is required to have ethereumjs shandong branch start off withdrawals at non zero block number after the merge.

This PR enhances the same
Geth genesis now carries a flag terminalTotalDifficultyPassed for this which indicates that the merge is before any other hardforks configured on non zero block number

codecov · 2022-11-17T10:52:25Z

Codecov Report

Merging #2427 (5c4f5d8) into master (161a402) will increase coverage by 0.01%.
The diff coverage is 95.65%.

Additional details and impacted files

Flag	Coverage Δ
block	`89.51% <ø> (ø)`
blockchain	`90.03% <ø> (ø)`
client	`87.47% <100.00%> (+0.01%)`	⬆️
common	`98.16% <100.00%> (ø)`
devp2p	`91.63% <ø> (+0.05%)`	⬆️
ethash	`∅ <ø> (∅)`
evm	`79.73% <95.00%> (+<0.01%)`	⬆️
rlp	`?`
statemanager	`89.46% <ø> (ø)`
trie	`90.36% <ø> (ø)`
tx	`97.80% <ø> (ø)`
util	`84.59% <ø> (ø)`
vm	`85.54% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

acolytec3 · 2022-11-17T15:05:46Z

packages/common/src/utils.ts

+    nonzeroIndex !== -1
+  ) {
+    // find index where block > 0
+    params.hardforks.splice(nonzeroIndex, 0, mergeConfig)


I don't think it's safe to assume the first nonzero hardfork will always be where the merge should go. We could very easily have an earlier hardfork (like ArrowGlacier/GrayGlacier which isn't even in this list) be the first nonzero one. Unless I'm not thinking clearly, we should always put Merge after London (if ArrowGlacier/GrayGlacier aren't specified), right?

ummm, doesn't actually matter in scheme of things if those are all at 0
the issue is when terminalTotalDifficultyPassed is false or not present, then where should we place Merge?

The order we assume is the order they showup in the config section, and it would have been nice to also have mergeBlock there even if its block is null

So that there is no confusion about the order, may be we can give this feedback to the geth guys and if the mergeBlock is not present in the config (like it doesn't for now) we can use this logic to place merge

K, this is the PR where they add it. I'm honestly not sure how to handle this one. The main place they reference this new parameter is here where the sync type is being decided (so akin to how we differentiate between full sync and beacon sync). Can you do beaconsync forward before the merge block or does the client have to continue with full sync till it hits the merge block? If the latter, I still think we should explicitly look for the london fork block number and insert this one after it. I can't think of a reason we should have GrayGlacier or ArrowGlacier in a custom network so we shouldn't need to check for them.

but the PoS or PoW consensus actually has nothing to do with pre/post london, it can be placed anywhere

ok, looking at the geths PR has got me confused about this flag, but we defintely need post merge hardforks differentiation which certainly needs merge, like withdrawals, and 4844. I am not clear if thats the way to go especially with mergeForkID confusion of it bein pre/post merge.
But since kiln, no network has had mergeForkId before merge so may be i am over thinking it?

ok i have tried to make this thing a bit more intelligenct, placing merge before the first postmerge hardfork, and is still dynamic with respect to mergeForkId block

Why is Geth not just giving a block number for the Merge?

All this dynamic handling and wild-guessing for this one time Merge event which has now passed seems totally overblown to me and doesn't really solve any problems any more (right?).

correct, may be they haven't had yet added handling for merge block
Also the genesis file seems weird, so the genesis block has difficulty and extraData belonging to PoA consensus, but shanghai is also at 0 for withdrawals so obviously merge hf has passed

@holgerd77 i think the latest implementation is logical:

place the merge hardfork before any merge dependent hardforks specified on non zero block as genesis can't be a PoS block.

Our dynamic handling of hardfork sequences and hardfork changes can handle the rest.

packages/common/src/utils.ts

acolytec3

See comments.

holgerd77 · 2022-11-22T10:47:52Z

Some general notes here: I generally find/found this Geth genesis file format only half-thought-through from the beginning, that was actually one main motivation why I introduced our own format within Common at the time.

So, my overall fear is always a bit that we are bringing in some clutter in our main (Common, partly other) code logic by trying to cope with this semi-defined structure from Geth genesis, particularly when new things are added there.

So in this case the (Merge) HF order should somewhat be extractable/readable from the Geth config file itself, since this is just some basic structural information piece one would need to build upon. Apparently it is not though, with these Merge-parameters extracted/separated from the other HF definitions.

Just some context/thinking from my side. Otherwise, long story short: since this PR only touches the Geth import logic in Common, I would be ok with merging.

Have I otherwise forgotten or overlooked something in my summary above or is there still some outstanding question which needs to be adressed? 🤔

g11tech · 2022-11-22T12:47:18Z

Some general notes here: I generally find/found this Geth genesis file format only half-thought-through from the beginning, that was actually one main motivation why I introduced our own format within Common at the time.

So, my overall fear is always a bit that we are bringing in some clutter in our main (Common, partly other) code logic by trying to cope with this semi-defined structure from Geth genesis, particularly when new things are added there.

So in this case the (Merge) HF order should somewhat be extractable/readable from the Geth config file itself, since this is just some basic structural information piece one would need to build upon. Apparently it is not though, with these Merge-parameters extracted/separated from the other HF definitions.

Just some context/thinking from my side. Otherwise, long story short: since this PR only touches the Geth import logic in Common, I would be ok with merging.

Have I otherwise forgotten or overlooked something in my summary above or is there still some outstanding question which needs to be adressed? thinking

Yes very valid points @holgerd77 , hopefully with this PR the merge hardfork placement is now very deterministic and is based on these factors:

Merge hardfork can't be placed at genesis
Place merge hf before any hardforks that require CL participation for e.g. withdrawals
Merge hardfork has to be placed just after genesis if any of the genesis hardforks make CL necessary for e.g. withdrawals

Have added the same in the comment in parseGeth..

acolytec3

I'm using these changes in my EIP4844 PR and it works like a charm with the latest changes.

holgerd77 · 2022-11-22T12:56:37Z

I'm using these changes in my EIP4844 PR and it works like a charm with the latest changes.

Ok, nice 🙂, I am also ok with merging here!

g11tech added PR state: WIP type: enhancement package: common target: master Work to be done towards master branch labels Nov 17, 2022

g11tech added PR state: merge ready and removed PR state: WIP labels Nov 17, 2022

g11tech changed the title ~~common: Set merge hardfork in parseGethParams before non zero blocks based on terminalTotalDifficultyPassed flag~~ common: Set merge hardfork in parseGethParams before hardforks scheduled post genesis if terminalTotalDifficultyPassed Nov 17, 2022

acolytec3 reviewed Nov 17, 2022

View reviewed changes

packages/common/src/utils.ts Outdated Show resolved Hide resolved

acolytec3 requested changes Nov 17, 2022

View reviewed changes

g11tech force-pushed the g11tech/fix-geth-genesis-parse branch 2 times, most recently from 055c9fb to 3aa4ad6 Compare November 19, 2022 07:57

g11tech changed the title ~~common: Set merge hardfork in parseGethParams before hardforks scheduled post genesis if terminalTotalDifficultyPassed~~ common: Parse post-merge hardfork config in parseGethParams and handle post-merge genesis block Nov 19, 2022

g11tech added 11 commits November 19, 2022 16:06

correctly set hf order

f0c6e4b

test specs

01b0ac1

fix the merge hf push condition

80e4f56

try placing merge block in better wat

421040a

fix test spec for kiln

d81ef6f

alternate fix of skipping only pos validation on genesis

d897836

move merge validations to non genesis

50b04ab

fix unsupported test case

3aa344a

correctly set hardfork

c27fc98

resolve test genesis poisioning

a9566e5

add comment

e54b1f7

g11tech force-pushed the g11tech/fix-geth-genesis-parse branch from b1d4a79 to e54b1f7 Compare November 19, 2022 10:37

g11tech added 4 commits November 19, 2022 17:35

cli arg for mergeforkid placement

4d125bd

place merge only after genesis

6b5d7df

fix tests

c5dd4c9

restore test

94b8c9d

g11tech added package: vm package: client package: evm labels Nov 21, 2022

g11tech added 3 commits November 22, 2022 17:54

address another case for merge just post genesis

5ae2e16

test case for the new edge case

b0f1092

add comment

5c4f5d8

acolytec3 approved these changes Nov 22, 2022

View reviewed changes

holgerd77 merged commit 33f68c6 into master Nov 22, 2022

holgerd77 deleted the g11tech/fix-geth-genesis-parse branch November 22, 2022 13:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

common: Parse post-merge hardfork config in parseGethParams and handle post-merge genesis block #2427

common: Parse post-merge hardfork config in parseGethParams and handle post-merge genesis block #2427

g11tech commented Nov 17, 2022

codecov bot commented Nov 17, 2022 •

edited

Loading

acolytec3 Nov 17, 2022

g11tech Nov 17, 2022

acolytec3 Nov 17, 2022

g11tech Nov 18, 2022

g11tech Nov 18, 2022

g11tech Nov 18, 2022

holgerd77 Nov 18, 2022

holgerd77 Nov 18, 2022

g11tech Nov 19, 2022

g11tech Nov 20, 2022

acolytec3 left a comment

holgerd77 commented Nov 22, 2022

g11tech commented Nov 22, 2022 •

edited

Loading

acolytec3 left a comment

holgerd77 commented Nov 22, 2022

common: Parse post-merge hardfork config in parseGethParams and handle post-merge genesis block #2427

common: Parse post-merge hardfork config in parseGethParams and handle post-merge genesis block #2427

Conversation

g11tech commented Nov 17, 2022

codecov bot commented Nov 17, 2022 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

acolytec3 left a comment

Choose a reason for hiding this comment

holgerd77 commented Nov 22, 2022

g11tech commented Nov 22, 2022 • edited Loading

acolytec3 left a comment

Choose a reason for hiding this comment

holgerd77 commented Nov 22, 2022

codecov bot commented Nov 17, 2022 •

edited

Loading

g11tech commented Nov 22, 2022 •

edited

Loading