Refactor v2

[johnwhitton]

No description provided.

[johnwhitton]

I’ve been looking at a problem with the deploy of the LightClient and narrowed it down to this line in RLPReader.sol
uint mask = 256 ** (WORD_SIZE - len) - 1;
When we are creating the rlpHeader the logsBloom is
"logsBloom": "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000”,
which has a length of 256 (this is for Ropsten, and localgeth)
Thus we were trying to go to a power of a negative number (which just failed without reverting or giving any error).
I put in a workaround of

            if (len >= WORD_SIZE) {
                len = WORD_SIZE - 1;
            }

And the deploy is now working
I noticed that the version of RLPReader we have was updated by Tuan Vu
I’m assuming it was copied originally from here
https://github.com/hamdiallam/Solidity-RLP/blob/master/contracts/RLPReader.sol#L351
This codebase was last modified on Jun 7 2021 and contracts have pragma solidity >=0.5.0 <0.7.0; we are using pragma solidity ^0.8.0;
Also the code from hamdiallam has some additional error handling which may have been applied after we made the original copy.
So there may be more cleanup work enhancements needed.
Currently the relay is still failing with Error: Transaction has been reverted by the EVM: latest log is here
Fix was applied here.

[johnwhitton]

Error calculating the seal when relaying blocks
Here is the current error when relaying a block after putting some logging in EthereumLightClient.sol
It is having trouble calling
Analysis
The block header for localgeth is as follows you can see sample blockHeaders from ropsten and ethereum here

header: {
    "parentHash": "0xdb9e3369509518820f176b86afc312ecafc54a4bc324491bd261df3614b55735",
    "uncleHash": "0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347",
    "coinbase": "0xf39fd6e51aad88f6f4ce6ab8827279cfffb92266",
    "stateRoot": "0xadfd9a8868580138c975dcaed43c6f56035eb9e89bd63cb5ac67651b32075733",
    "transactionsTrie": "0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421",
    "receiptTrie": "0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421",
    "logsBloom": "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
    "difficulty": "0x20000",
    "number": "0x2",
    "gasLimit": "0x7a4f0e",
    "gasUsed": "0x0",
    "timestamp": "0x62a8d530",
    "extraData": "0xd983010a12846765746888676f312e31382e328664617277696e",
    "mixHash": "0x233714ae9205bb1305214026b9335b113203dc51d4e68a876400c8936795f8d4",
    "nonce": "0x76a1124da977aa93",
    "bloom": "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"
}

EthereumParser.sol iterates through the header fields to calculate the sealRlpData

        bytes[] memory rlpFields = new bytes[](14);
        RLPReader.Iterator memory it = _rlpHeader.toRlpItem().iterator();
        uint256 idx = 0;
        while (it.hasNext() && idx < 14) {
            console.log("idx: ", idx);
            if (idx == 13) {
                it.next();
                it.next();
            }
            rlpFields[idx] = it.next().toRlpBytes();
            idx++;
        }

The goal of the logic above is to ignore the following lines in the header file

    "baseFeePerGas": "0x7",
    "baseFee": "0x7",

However for localgeth those lines are omitted (as with some of the earlier blocks in Ropsten.
The logic should be modified to conditionally skip the lines above only if they are populated.
This could be done by calculating the length of the header rows and replacing the hardocded skipping on idx ==13
The other option is to modify localgeth genesis.json to force it to add in the baseFee information.

Command
yarn cli ethRelay relay http://localhost:8645 http://localhost:8545 0x017f8C7d1Cb04dE974B8aC1a6B8d3d74bC74E7E1 -d ./src/cli/.dag
Error

console.log:
    calling _pruneBlocks
    checking blockExists
    Parsing BlockHeader
    checking Parentblock
    checking block height
    checking timestamp
    checking difficulty
    verifying block POW

  Error: Transaction reverted without a reason string
      at EthereumLightClient.next (contracts/lib/RLPReader.sol:33)
      at EthereumLightClient.calcBlockSealHash (contracts/EthereumParser.sol:98)
      at EthereumLightClient.addBlockHeader (contracts/EthereumLightClient.sol:178)

[johnwhitton]

Error validating Epoch
After resolving the above we now get Error: VM Exception while processing transaction: reverted with reason string 'epoch out of range!'

Analysis
For local testing epoch 0 is being used.
When verifying the EthHash in ethhash.sol we call bytes32 rootHash = getRootHash(uint64(epoch));
Which calls getRootHash in MerkleRoot.sol which has hardcoded these values

contract MerkelRoots {
    uint64 constant epochStart = 411;
    uint64 constant epochEnd = 411;
    bytes constant ROOTS = "\xf8\x7d\x60\x5d\xd4\xbd\xaf\xc3\x9b\x13\xb4\x5b\x6a\xbf\x6b\x92\x11\x96\xb0\xf3\xb5\xd7\xc5\x1b\x31\x82\x06\x64\x10\x4a\x69\x7d";

   function getRootHash(uint64 epoch) internal pure returns(bytes32 hash) {
       bytes memory roots = ROOTS;
       require(epoch >= epochStart && epoch <= epochEnd, "epoch out of range!");
       uint256 index = epoch - epochStart + 1; // skip length
       assembly{
           hash := mload(add(roots, mul(index, 0x20)))
       }
   }
}
**Workaround was to change `uint64 constant epochStart = 0;`**
Needs further analysis as hot `ROOTS` is generated
`root: "0x2c2fcc0735c171e57f5fa49f0d3b30f777fa2eea3531f418797af14b1a98d9b8"`
so we can make the following change

// Test data for Ropsten
// uint64 constant epochStart = 0;
// uint64 constant epochEnd = 411;
// bytes constant ROOTS =
//     "\xf8\x7d\x60\x5d\xd4\xbd\xaf\xc3\x9b\x13\xb4\x5b\x6a\xbf\x6b\x92\x11\x96\xb0\xf3\xb5\xd7\xc5\x1b\x31\x82\x06\x64\x10\x4a\x69\x7d";
// Configuration for LocalGeth
uint64 constant epochStart = 0;
uint64 constant epochEnd = 411;
bytes constant ROOTS =
    "\x2c\x2f\xcc\x07\x35\xc1\x71\xe5\x7f\x5f\xa4\x9f\x0d\x3b\x30\xf7\x77\xfa\x2e\xea\x35\x31\xf4\x18\x79\x7a\xf1\x4b\x1a\x98\xd9\xb8";

After changing this we then received an error using `hashimoto` analsyis below

**Command**
yarn cli ethRelay relay http://localhost:8645 http://localhost:8545 0x6a754B8573195AA9Dfea97eEe704C981eCf2c1FF -d ./src/cli/.dag

**Log**
Error: VM Exception while processing transaction: reverted with reason string 'epoch out of range!'
      at EthereumLightClient.whenNotPaused (@openzeppelin/contracts-upgradeable/security/PausableUpgradeable.sol:57)
      at EthereumLightClient.getRootHash (contracts/ethash/MerkelRoot.sol:11)
      at EthereumLightClient.verifyEthash (contracts/ethash/ethash.sol:493)
      at EthereumLightClient.addBlockHeader (contracts/EthereumLightClient.sol:179)
      at async HardhatNode.runCall (/Users/john/one-wallet/horizon/node_modules/hardhat/src/internal/hardhat-network/provider/node.ts:616:20)
      at async EthModule._callAction (/Users/john/one-wallet/horizon/node_modules/hardhat/src/internal/hardhat-network/provider/modules/eth.ts:354:9)

[johnwhitton]

Error with hashimoto calculation

Analysis
The error occurs when generating the header hash and nonce using the hashimoto algorithm.
It occurs here return (a * 0x01000193) ^ b;
And is passed these values return (3789160042 * 0x01000193) ^ 3789160042
The error occurs on this line uint32 parent = fnv(i ^ seedHead, mix[i % mix.length]) % rows;
Below are the Logged Values

    hashimoto i:  0
    mix.length:  32
    rows:  8388607
    seedHead:  3789160042
    a:  3789160042
    b:  3789160042

Command

 yarn cli ethRelay relay http://localhost:8645 http://localhost:8545 0x6a754B8573195AA9Dfea97eEe704C981eCf2c1FF -d ./src/cli/.dag

Log

 Error: VM Exception while processing transaction: reverted with panic code 0x11 (Arithmetic operation underflowed or overflowed outside of an unchecked block)
      at EthereumLightClient.fnv (contracts/ethash/ethash.sol:44)
      at EthereumLightClient.hashimoto (contracts/ethash/ethash.sol:438)
      at EthereumLightClient.verifyEthash (contracts/ethash/ethash.sol:495)
      at EthereumLightClient.addBlockHeader (contracts/EthereumLightClient.sol:183)

References

• Mining Algorithms
• Dagger Hashimoto
• FNV hash algorithm
• GO etchash algorithm.go

[johnwhitton]

Am currently looking at whether the root cause of all the above is related to the fact that the codebase has been built specifically to align with the current ethereum mainnet.

To replicate this I have modified the genesis json to include additional configuration parameters related to london as mentioned here and here.

By modifying genesis.config as follows (the key line "londonBlock": 0,)

{
    "config": {
        "chainId": 8788,
        "homesteadBlock": 0,
        "eip150Block": 0,
        "eip155Block": 0,
        "eip158Block": 0,
        "byzantiumBlock": 0,
        "constantinopleBlock": 0,
        "petersburgBlock": 0,
        "istanbulBlock": 0,
        "muirGlacierBlock": 0,
        "berlinBlock": 0,
        "londonBlock": 0,
        "arrowGlacierBlock": 0,
        "grayGlacierBlock": 0,
        "mergeNetsplitBlock": 0,
        "ethash": {}
    },
    "difficulty": "1",
    "gasLimit": "8000000",
    "alloc": {
        "9Cf3000a2f555607Ce330Bf70B0f67c7cBcd3be8": {
            "balance": "300000"
        }
    }
}

we now get our block headers including basFee and baseFeePerGas.

 {
    "parentHash": "0x73736e2208cff63f17f5a0af53e78b16df24dbb439633f137300cfc002a468a1",
    "uncleHash": "0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347",
    "coinbase": "0xf39fd6e51aad88f6f4ce6ab8827279cfffb92266",
    "stateRoot": "0x3fad6519e1f95df7fe10feb4eeab0c49c8edde320338ecc0000184b563299dcb",
    "transactionsTrie": "0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421",
    "receiptTrie": "0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421",
    "logsBloom": "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
    "difficulty": "0x231a8",
    "number": "0xc8",
    "gasLimit": "0x94616f",
    "gasUsed": "0x0",
    "timestamp": "0x62abc5ae",
    "extraData": "0xd983010a12846765746888676f312e31382e328664617277696e",
    "mixHash": "0x0a40a3c6f22e86feb2b3d72af550959669c76a0001b3f907ae14d868ce0f8daa",
    "nonce": "0x113fb9eda7492a17",
    "baseFeePerGas": "0x7",
    "baseFee": "0x7",
    "bloom": "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"
}

Currently have begun creating the DAG for the new chain, then will update the initData to reflect the new configuration.
Will also update "MerkleRoot.sol` to have the corrrect root for localgeth and epochStart and epochEnd set to 0.

[johnwhitton]

Have updated genesis.json and initData and we now generate the headers similar to that of Ethereum mainnet.
Also updated 'MerkleRoot.sol' to use epoch 0 and placed in the encode root retrieved from proofs.root.
This resolved the issues with calculating the seal and validating the epoch.
However the issue with the hashimota calculation still exists.
Will review further tomorrow.

[johnwhitton]

Latest log from debugging verifyEthHash callling hashimoto in ethhash.sol
console.log:
sealHash: 74390009781685601618541290251960722186319639099107416850618130334421248486748
header.nonce: 1840997502332555572
header.number: 1001
header.difficulty: 210447
header.mixHash: 46150766055922273093573656953585866175828334889140758841637818972486362424218
hash: 0xa47740e251bacf5ebec2ae7083b98b0c6781b5957d4c550ff3fc043dcf2dad5c
nonce: 1840997502332555572
number: 1001
difficulty: 210447
mixHash: 46150766055922273093573656953585866175828334889140758841637818972486362424218
epoch: 0
rootHash: 0x2c2fcc0735c171e57f5fa49f0d3b30f777fa2eea3531f418797af14b1a98d9b8
size: 1073741696 //calculated using getFullSize
size: 1073741696
mixBytes: 128 //constant
rows: 8388607. //size divided by mixBytes
seedBeforeSha: 0xa47740e251bacf5ebec2ae7083b98b0c6781b5957d4c550ff3fc043dcf2dad5c34d1e96e79898c19 //seed first 32 bytes is the hash and then nonce
seed: 0x4dee06e420c1601d8ffd84a241e87833bdf5ffa42c379816e91139ff1e28958ee13fce99bc8156f8116233d1de5f9ee8e0d24eb1f69a12253b3a09e35d6ae58f // seedBeforeSha.sha3_512();
seedHead: 3825659469 // uint32 seedHead = seed.Uint32(0);
mix i: 0
mix[i]: 3825659469
mix i: 1
mix[i]: 492880160
mix i: 2
mix[i]: 2726624655
mix i: 3
mix[i]: 863561793
mix i: 4
mix[i]: 2768238013
mix i: 5
mix[i]: 379074348
mix i: 6
mix[i]: 4281930217
mix i: 7
mix[i]: 2392139806
mix i: 8
mix[i]: 2580430817
mix i: 9
mix[i]: 4166418876
mix i: 10
mix[i]: 3509805585
mix i: 11
mix[i]: 3902693342
mix i: 12
mix[i]: 2974733024
mix i: 13
mix[i]: 621976310
mix i: 14
mix[i]: 3809032763
mix i: 15
mix[i]: 2414176861
mix i: 16
mix[i]: 3825659469
mix i: 17
mix[i]: 492880160
mix i: 18
mix[i]: 2726624655
mix i: 19
mix[i]: 863561793
mix i: 20
mix[i]: 2768238013
mix i: 21
mix[i]: 379074348
mix i: 22
mix[i]: 4281930217
mix i: 23
mix[i]: 2392139806
mix i: 24
mix[i]: 2580430817
mix i: 25
mix[i]: 4166418876
mix i: 26
mix[i]: 3509805585
mix i: 27
mix[i]: 3902693342
mix i: 28
mix[i]: 2974733024
mix i: 29
mix[i]: 621976310
mix i: 30
mix[i]: 3809032763
mix i: 31
mix[i]: 2414176861
mix.length: 32

rootHash: 0x2c2fcc0735c171e57f5fa49f0d3b30f777fa2eea3531f418797af14b1a98d9b8
loopAccess i 0
Calling fnv
iseedHead:  3825659469
mixer:  3825659469
a:  3825659469
b:  3825659469

Error: VM Exception while processing transaction: reverted with panic code 0x11 (Arithmetic operation underflowed or overflowed outside of an unchecked block)
at EthereumLightClient.fnv (contracts/ethash/ethash.sol:48)
at EthereumLightClient.hashimoto (contracts/ethash/ethash.sol:457)
at EthereumLightClient.verifyEthash (contracts/ethash/ethash.sol:524)
at EthereumLightClient.addBlockHeader (contracts/EthereumLightClient.sol:177)

[johnwhitton]

Relayer is now working
Summary of changes

• Modification of localgeth to use london fork (to align with header info from Ethereum)
• Modification of ethhash.sol to handle unchecked

   function fnv(uint32 a, uint32 b) internal view returns (uint32) {
        unchecked {
            return (a * 0x01000193) ^ b;
        }
    }

• Modification of MerkleRoot.sol to add in localgeth DAG info

contract MerkelRoots {
    // Ropsten DAG and epoch information
    // uint64 constant epochStart = 411;
    // uint64 constant epochEnd = 411;
    // bytes constant ROOTS =
    //     "\xf8\x7d\x60\x5d\xd4\xbd\xaf\xc3\x9b\x13\xb4\x5b\x6a\xbf\x6b\x92\x11\x96\xb0\xf3\xb5\xd7\xc5\x1b\x31\x82\x06\x64\x10\x4a\x69\x7d";

    // LocalGeth DAG and epoch information
    uint64 constant epochStart = 0;
    uint64 constant epochEnd = 0;
    bytes constant ROOTS =
        "\x2c\x2f\xcc\x07\x35\xc1\x71\xe5\x7f\x5f\xa4\x9f\x0d\x3b\x30\xf7\x77\xfa\x2e\xea\x35\x31\xf4\x18\x79\x7a\xf1\x4b\x1a\x98\xd9\xb8";

    function getRootHash(uint64 epoch) internal pure returns (bytes32 hash) {
        bytes memory roots = ROOTS;
        require(
            epoch >= epochStart && epoch <= epochEnd,
            "epoch out of range!"
        );
        uint256 index = epoch - epochStart + 1; // skip length
        assembly {
            hash := mload(add(roots, mul(index, 0x20)))
        }
    }
}

• Modification of 004_deplopy_ethereumLightClient.ts to use blocknum 1000 (initChainData has approx 3000 blocks
  const blockNum = 1000 // Localgeth Block Number

Working Log

johnlaptop horizon (refactorV2) $ yarn cli ethRelay relay http://localhost:8645 http://localhost:8545 0x76420bfC35CD4841C5dd3920f3a0f0aC4831208A -d ./src/cli/.dag
yarn run v1.22.18
warning ../../package.json: No license field
$ node src/cli/index.js ethRelay relay http://localhost:8645 http://localhost:8545 0x76420bfC35CD4841C5dd3920f3a0f0aC4831208A -d ./src/cli/.dag
ELC last block number: 1000
block to relay: 1001
header hash 0x479a245019d7bed120b30e6dd40e75d34f8cad600138ed9d16432daa981cd4b2
headerHash:  a47740e251bacf5ebec2ae7083b98b0c6781b5957d4c550ff3fc043dcf2dad5c
proofs.root: 0x2c2fcc0735c171e57f5fa49f0d3b30f777fa2eea3531f418797af14b1a98d9b8
new block number: 1001
ELC last block number: 1001
block to relay: 1002
header hash 0xbed2ffb165399c77a9e702d5424e76fc1b93d629dc03f9b843d154264715b965
headerHash:  86143fe67e6c3bdbc2a8aa3bd4cd9f42fa7474215c51cb5b1a7c1803eb4d69a2
proofs.root: 0x2c2fcc0735c171e57f5fa49f0d3b30f777fa2eea3531f418797af14b1a98d9b8
new block number: 1002
ELC last block number: 1002
block to relay: 1003
header hash 0xc91f5e6f1b9928d501b7d7a20db4d36bf9163095f88537dde630c5a5fe1ed4eb
headerHash:  76ca9edc1982ed7007f2fdd0ab82589bb2104f43fef79ad53846bc1b6802e759
proofs.root: 0x2c2fcc0735c171e57f5fa49f0d3b30f777fa2eea3531f418797af14b1a98d9b8
new block number: 1003
ELC last block number: 1003
block to relay: 1004
header hash 0xf14df3dd0d891613d636a9a9233ce6f77f1760e716bdc660b43d24cc3d4d7664
headerHash:  5407f3453605a015bcd6a37869a05956e56289e96ee9e247efacbfae0d34f60a
proofs.root: 0x2c2fcc0735c171e57f5fa49f0d3b30f777fa2eea3531f418797af14b1a98d9b8
new block number: 1004
ELC last block number: 1004

[johnwhitton]

Completed these tasks

• Add dependency for LightClient to TokenLocker on Deploy
• Create backup branch refactorBeforeProxyChanges
• Update deploys to use proxies
• Remove hardcoded deploy variables (use config.js instead)
• Remove deploy tasks from lib and place under deploy

For more on the deployment approach and use of plugins see here

[johnwhitton]

Completed these tasks

• Clean up lib (move deploys to deploy and tests to test and remove upgrade)
• Upgrade isomorphic-rpc yarn upgrade isomorphic-rpc@1.2.2 to fix node:53398) UnhandledPromiseRejectionWarning: Error: {"jsonrpc":"2.0","id":1,"error":{"code":-32601,"message":"the method toJSON does not exist/is not available"}}
• Move from web3 to ethers
• make consistent use of ethers.js replacing libraries such as web3, rlp, bigNumber, ethereumjs-util. Recommendation is ethers vs web3 based on hardhat integration this article, these stats and these comments Ethers.js loads slightly faster thanks to its noticeably smaller size, which may offer better performance. and However, the blockchain industry as a whole is slowly migrating towards a younger alternative – Ethers.js.
• Consolidate helpers for cli and lib (create one js file per are in cli if needed e.g. bridge.js)
  • src/cli/bridge => src/bridge
  • src/cli/elc => src/elc.js
  • src/cli/eth2hmy-relay => src/eth2hmy-relay
  • src/cli/ethashProof => src/ethashProof (this is DAG related)
  • src/cli/verifier => src/eprover
  • src/cli/lib => lib (also update ethWeb3 and hmyWeb3 to use ethers)

Next task is here

• Write bridge.js proof calls (currently passing incorrect parameters to Harmony TokenLocker validateAndExecuteProof)