Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

blockchain: allow optimistic block insertion in blockchain #3584

Draft
wants to merge 10 commits into
base: master
Choose a base branch
from
Draft

blockchain: allow optimistic block insertion in blockchain #3584

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
3b24f64
blockchain: allow optimistic block insertion in blockchain
g11tech Aug 13, 2024
a69d63f
add and use optimistic number to hash key/dbop to avoid conflict with…
g11tech Aug 14, 2024
e640849
add and handle further modifications and flags to handle skeleton fun…
g11tech Aug 17, 2024
bbb140b
debug and fix skeleton tests
g11tech Aug 18, 2024
23fd13f
further debug and fix the beacon sync sim test and get it working
g11tech Aug 18, 2024
c7f2ae2
refactor the entire approach to simply allow building normal index as…
g11tech Aug 26, 2024
d9e3297
modify putopts to be more understanable and add the optimistic block …
g11tech Sep 2, 2024
7bd20c2
add 4444 and optimistic spec debug and get it working
g11tech Sep 3, 2024
e4253d6
add 4444 and optimisitc examples
g11tech Sep 14, 2024
3bbed08
remove skipping validation
g11tech Sep 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
85 changes: 85 additions & 0 deletions packages/blockchain/examples/4444.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
import { createBlock } from '@ethereumjs/block'
import { createBlockchain } from '@ethereumjs/blockchain'
import { Common, Hardfork, Mainnet } from '@ethereumjs/common'
import { bytesToHex, hexToBytes } from '@ethereumjs/util'

const main = async () => {
const common = new Common({ chain: Mainnet, hardfork: Hardfork.Shanghai })
// Use the safe static constructor which awaits the init method
const blockchain = await createBlockchain({
common,
})

// We are using this to create minimal post merge blocks between shanghai and cancun in line with the
// block hardfork configuration of mainnet
const chainTTD = BigInt('58750000000000000000000')
const shanghaiTimestamp = 1681338455

// We use minimal data to provide a sequence of blocks (increasing number, difficulty, and then setting parent hash to previous block)
const block1 = createBlock(
{
header: {
// 15537393n is terminal block in mainnet config
number: 15537393n + 500n,
// Could be any parenthash other than 0x00..00 as we will set this block as a TRUSTED 4444 anchor
// instead of genesis to build blockchain on top of. One could use any criteria to set a block
// as trusted 4444 anchor
parentHash: hexToBytes(`0x${'20'.repeat(32)}`),
timestamp: shanghaiTimestamp + 12 * 500,
},
},
{ common, setHardfork: true },
)
const block2 = createBlock(
{
header: {
number: block1.header.number + 1n,
parentHash: block1.header.hash(),
timestamp: shanghaiTimestamp + 12 * 501,
},
},
{ common, setHardfork: true },
)
const block3 = createBlock(
{
header: {
number: block2.header.number + 1n,
parentHash: block2.header.hash(),
timestamp: shanghaiTimestamp + 12 * 502,
},
},
{ common, setHardfork: true },
)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: could these be packed in a for loop, and the "magic numbers" (500) be put as a constant?


let headBlock, blockByHash, blockByNumber
headBlock = await blockchain.getCanonicalHeadBlock()
console.log(
`Blockchain ${blockchain.consensus.algorithm} Head: ${headBlock.header.number} ${bytesToHex(headBlock.hash())}`,
)
// Blockchain casper Head: 0 0xd4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3

// 1. We can put any post merge block as 4444 anchor by using TTD as parentTD
// 2. For pre-merge blocks its prudent to supply correct parentTD so as to respect the
// hardfork configuration as well as to determine the canonicality of the chain on future putBlocks
await blockchain.putBlock(block1, { parentTd: chainTTD })
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still do not fully understand this anchor concept. What is anchored and how?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

block1 is the 4444 anchor i.e. the parent chain to block1 doesn't need to be added to the blockchain and blockchain can be build forward from here

it acts as a re-genesis anchor, one specifies an anchor by providing parentTD, for a PoS block parentTD can just be chainTTD, for a PoW block an actual parentTD should be added so as to eventually and correctly matchup with the terminal block TTD when forward chain blocks will be added on top of it

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are one of the things which are too implicit, at least on the API, maybe the DB level. I am still not fully grasping if the parentTd is also needed (for calculating/validating something) or is only set in some way to indicate the anchor status.

No matter how it is: this is unnecessarily implicit and we should think about the "anchor" (or call it checkpoint?) status and give this its own API and database representation.

I think "Checkpoint" might actually be the more common and established terminolgy.

So I think we definitely want to have some external relation in the DB where we store all the checkpoints (and not only have this in some very implicit form), and can e.g. extend our API with things like:

blockchain.getCheckpoints()
blockchain.isCheckpoint(5n)

(or first there would be the question if there can be only one of these, but guess there can be several respectively would make sense?)

(wonder if we can even lign in the genesis block in this concept, guess this is exactly the same as a checkpoint?)

And then for setting these checkpoints, this should be very explicitly named. So if parentTd is still necessary, even if we store a Checkpoints -> [ Block Numbers ] relation, then this should minimally be provided with naming the option setCheckpoint: 560934n (and so provide the parentTD still, but set the very emphasis on this checkpoint setting, so one can read from the API call what is happening here).

headBlock = await blockchain.getCanonicalHeadBlock()
console.log(
`Blockchain ${blockchain.consensus.algorithm} Head: ${headBlock.header.number} ${bytesToHex(headBlock.hash())}`,
)
// Blockchain casper Head: 15537893 0x26cb3bfda75027016c17d737fdabe56f412925311b42178a675da88a41bbb7e7

await blockchain.putBlock(block2)
headBlock = await blockchain.getCanonicalHeadBlock()
console.log(
`Blockchain ${blockchain.consensus.algorithm} Head: ${headBlock.header.number} ${bytesToHex(headBlock.hash())}`,
)
// Blockchain casper Head: 15537894 0x6c33728cd8aa21db683d94418fec1f7ee1cfdaa9b77781762ec832da40ec3a7c

await blockchain.putBlock(block3)
headBlock = await blockchain.getCanonicalHeadBlock()
console.log(
`Blockchain ${blockchain.consensus.algorithm} Head: ${headBlock.header.number} ${bytesToHex(headBlock.hash())}`,
)
// Blockchain casper Head: 15537895 0x4263ec367ce44e4092b79ea240f132250d0d341639afbaf8c0833fbdd6160d0f
}
void main()
141 changes: 141 additions & 0 deletions packages/blockchain/examples/optimistic.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,141 @@
import { createBlock } from '@ethereumjs/block'
import { createBlockchain } from '@ethereumjs/blockchain'
import { Common, Hardfork, Mainnet } from '@ethereumjs/common'
import { bytesToHex, hexToBytes } from '@ethereumjs/util'

const main = async () => {
const common = new Common({ chain: Mainnet, hardfork: Hardfork.Shanghai })
// Use the safe static constructor which awaits the init method
const blockchain = await createBlockchain({
common,
})

// We are using this to create minimal post merge blocks between shanghai and cancun in line with the
// block hardfork configuration of mainnet
const chainTTD = BigInt('58750000000000000000000')
const shanghaiTimestamp = 1681338455

// We use minimal data to construct random block sequence post merge/paris to worry not much about
// td's pre-merge while constructing chain
const block1 = createBlock(
{
header: {
// 15537393n is terminal block in mainnet config
number: 15537393n + 500n,
// Could be any parenthash other than 0x00..00 as we will set this block as a TRUSTED 4444 anchor
// instead of genesis to build blockchain on top of. One could use any criteria to set a block
// as trusted 4444 anchor
parentHash: hexToBytes(`0x${'20'.repeat(32)}`),
timestamp: shanghaiTimestamp + 12 * 500,
},
},
{ common, setHardfork: true },
)
const block2 = createBlock(
{
header: {
number: block1.header.number + 1n,
parentHash: block1.header.hash(),
timestamp: shanghaiTimestamp + 12 * 501,
},
},
{ common, setHardfork: true },
)
const block3 = createBlock(
{
header: {
number: block2.header.number + 1n,
parentHash: block2.header.hash(),
timestamp: shanghaiTimestamp + 12 * 502,
},
},
{ common, setHardfork: true },
)

let headBlock, blockByHash, blockByNumber
headBlock = await blockchain.getCanonicalHeadBlock()
console.log(
`Blockchain ${blockchain.consensus.algorithm} Head: ${headBlock.header.number} ${bytesToHex(headBlock.hash())}`,
)
// Blockchain casper Head: 0 0xd4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3

// allows any block > head + 1 to be put as non canonical block
await blockchain.putBlock(block3, { canonical: true })
headBlock = await blockchain.getCanonicalHeadBlock()
blockByHash = await blockchain.getBlock(block3.hash()).catch((e) => null)
blockByNumber = await blockchain.getBlock(block3.header.number).catch((e) => null)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We expect that this does not throw - right? So the catch should be removed? (same for blockByHash)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, its just a pattern used to log the status, sometimes blockByNumber will not be found depending upon the scenario

console.log(
`putBlock ${block3.header.number} ${bytesToHex(block3.hash())} byHash: ${blockByHash ? true : false} byNumber: ${blockByNumber ? true : false} headBlock=${headBlock.header.number}`,
)
// putBlock 15537895 0x4263ec367ce44e4092b79ea240f132250d0d341639afbaf8c0833fbdd6160d0f byHash: true byNumber: true headBlock=0

let hasBlock1, hasBlock2, hasBlock3
hasBlock1 = (await blockchain.getBlock(block1.header.number).catch((e) => null)) ? true : false
hasBlock2 = (await blockchain.getBlock(block2.header.number).catch((e) => null)) ? true : false
hasBlock3 = (await blockchain.getBlock(block3.header.number).catch((e) => null)) ? true : false
console.log(
`canonicality: head=${headBlock.header.number}, 0 ... ${block1.header.number}=${hasBlock1} ${block2.header.number}=${hasBlock2} ${block3.header.number}=${hasBlock3} `,
)
// canonicality: head=0, 0 ... 15537893=false 15537894=false 15537895=true

await blockchain.putBlock(block2, { canonical: true })
headBlock = await blockchain.getCanonicalHeadBlock()
blockByHash = await blockchain.getBlock(block2.hash()).catch((e) => null)
blockByNumber = await blockchain.getBlock(block2.header.number).catch((e) => null)
console.log(
`putBlock ${block2.header.number} ${bytesToHex(block2.hash())} byHash: ${blockByHash ? true : false} byNumber: ${blockByNumber ? true : false} headBlock=${headBlock.header.number}`,
)
// putBlock 15537894 0x6c33728cd8aa21db683d94418fec1f7ee1cfdaa9b77781762ec832da40ec3a7c byHash: true byNumber: true headBlock=0

hasBlock1 = (await blockchain.getBlock(block1.header.number).catch((e) => null)) ? true : false
hasBlock2 = (await blockchain.getBlock(block2.header.number).catch((e) => null)) ? true : false
hasBlock3 = (await blockchain.getBlock(block3.header.number).catch((e) => null)) ? true : false
console.log(
`canonicality: head=${headBlock.header.number}, 0 ... ${block1.header.number}=${hasBlock1} ${block2.header.number}=${hasBlock2} ${block3.header.number}=${hasBlock3} `,
)
// canonicality: head=0, 0 ... 15537893=false 15537894=true 15537895=true

// 1. We can put any post merge block as 4444 anchor by using TTD as parentTD
// 2. For pre-merge blocks its prudent to supply correct parentTD so as to respect the
// hardfork configuration as well as to determine the canonicality of the chain on future putBlocks
await blockchain.putBlock(block1, { parentTd: chainTTD })
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I were to validateBlocks = true and valdidateConsensus = true, what should happen if I put canonical = true here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

blockvalidation depends on parent availability in the blockchain and not on any other factor, for optimistic blocks it doesn't happen.

so lets say this was an optimistic block i.e. its parent was not in blockchain, validate blocks will be skipped. if you add canonical=true here, it will add the number => hash index and will run the head update rules (depending on if this is pow block or pos block) since parentTd has been specified and it will assume that it parent chain is confirmed and doesn't need to be added to blockchain

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

blockvalidation depends on parent availability in the blockchain

This is too implicit behavior. If we have set validateBlocks = true, blocks should be validated. Period. There shouldn't be inner auto-triggered exceptions from this rule (except maybe for the very explicit task of setting a checkpoint), otherwise one could not rely on the blockchain functionality anymore.

If one wants to deactivate block validation, we should add an option blockchain.setBlockValidation(false) or similar.

Guess this should then be accompanied by a flag for blockchain (if we go this route) to indicate that we might deal with a not-fully validated blockchain.

At least all these things (the "state" of the blockchain) should be transparent and requestable, and not just be "implicitly there".

headBlock = await blockchain.getCanonicalHeadBlock()
hasBlock1 = (await blockchain.getBlock(block1.header.number).catch((e) => null)) ? true : false
hasBlock2 = (await blockchain.getBlock(block2.header.number).catch((e) => null)) ? true : false
hasBlock3 = (await blockchain.getBlock(block3.header.number).catch((e) => null)) ? true : false
console.log(
`canonicality: head=${headBlock.header.number}, 0 ... ${block1.header.number}=${hasBlock1} ${block2.header.number}=${hasBlock2} ${block3.header.number}=${hasBlock3} `,
)
// canonicality: head=15537893, 0 ... 15537893=true 15537894=true 15537895=true

await blockchain.putBlock(block2)
headBlock = await blockchain.getCanonicalHeadBlock()
console.log(
`Blockchain ${blockchain.consensus.algorithm} Head: ${headBlock.header.number} ${bytesToHex(headBlock.hash())}`,
)
// Blockchain casper Head: 15537894 0x6c33728cd8aa21db683d94418fec1f7ee1cfdaa9b77781762ec832da40ec3a7c

hasBlock1 = (await blockchain.getBlock(block1.header.number).catch((e) => null)) ? true : false
hasBlock2 = (await blockchain.getBlock(block2.header.number).catch((e) => null)) ? true : false
hasBlock3 = (await blockchain.getBlock(block3.header.number).catch((e) => null)) ? true : false
console.log(
`canonicality: head=${headBlock.header.number}, 0 ... ${block1.header.number}=${hasBlock1} ${block2.header.number}=${hasBlock2} ${block3.header.number}=${hasBlock3} `,
)
// canonicality: head=15537894, 0 ... 15537893=true 15537894=true 15537895=true

await blockchain.putBlock(block3)
headBlock = await blockchain.getCanonicalHeadBlock()
console.log(
`Blockchain ${blockchain.consensus.algorithm} Head: ${headBlock.header.number} ${bytesToHex(headBlock.hash())}`,
)
// Blockchain casper Head: 15537895 0x4263ec367ce44e4092b79ea240f132250d0d341639afbaf8c0833fbdd6160d0f

hasBlock1 = (await blockchain.getBlock(block1.header.number).catch((e) => null)) ? true : false
hasBlock2 = (await blockchain.getBlock(block2.header.number).catch((e) => null)) ? true : false
hasBlock3 = (await blockchain.getBlock(block3.header.number).catch((e) => null)) ? true : false
console.log(
`canonicality: head=${headBlock.header.number}, 0 ... ${block1.header.number}=${hasBlock1} ${block2.header.number}=${hasBlock2} ${block3.header.number}=${hasBlock3} `,
)
// canonicality: head=15537895, 0 ... 15537893=true 15537894=true 15537895=true
}
void main()
Loading
Loading