Fix pyspec hive tests

[jochem-brouwer]

This PR fixes the pyspec hive tests. These tests are tests from the execution-spec-tests repo and re-execute these tests, but now using the engine API of the client.

No consensus bugs were found. However, some missing features and some annoying stuff has been fixed in this PR: (most of those are edge cases, except genesis files where nonce is nonzero for some accounts)

• When importing genesis state (for instance, from a Geth genesis), also read the nonce (if available)
• • This caused some pyspec tests not to run, because the provided state after putting this genesis has a different state root than what the genesis block actually points to (which results in client reporting: parent block not executed yet)
• Ensures that if genesis blocks have very large (larger than Number.MAX_SAFE_INTEGER) gasLimit or difficulty, this is parsed right by using BigInt
• Ensures in Trie that if we request if the empty state root (so 0 accounts, not even empty ones) is available, it returns true (this value is not in the DB). Fixes a test which starts with empty genesis allocation.
• Fixes setting up the genesis trie db in VM where, if you put empty accounts, this account is actually in the touched list and thus deleted on the first tx run (only exception of course if this tx creates a non-empty account)
• Updates engine-only RPC to only require the strict methods which should be available by current spec
• • Found "bugs" on hive, where some methods which are not required were called on the engine-only port: simulators/ethereum/engine: send eth_ requests to port 8545 ethereum/hive#752
• Fixes a nasty instance where you create a private key file which ends with a newline (private key import then thinks private key is 33 bytes instead of expected 32 bytes). Now client removes all newlines from the private key file.

Before this PR: fail 93/94. Current: fail 0/94

To run:

• Ensure you have updated hive to the latest version
• From hive: go run hive.go --docker.output --client ethereumjs --sim ethereum/pyspec

[codecov]

Codecov Report

❗ No coverage uploaded for pull request base (develop-v7@a42c10e). Click here to learn what that means.
The diff coverage is n/a.

Additional details and impacted files

Flag	Coverage Δ
ethash	∅ <0.00%> (?)
evm	79.48% <0.00%> (?)
tx	94.45% <0.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

[jochem-brouwer]

Current status:

[b8fa5a84] failing tests:
[b8fa5a84] ethereumjs/withdrawals/withdrawals/use_value_in_contract/000/shanghai: payload status mismatch
[b8fa5a84] ethereumjs/eips/eip3860/initcode_limit_create2_opcode/000/max_size_zeros/shanghai: payload status mismatch
[b8fa5a84] ethereumjs/eips/eip3860/initcode_limit_create2_opcode/001/max_size_ones/shanghai: payload status mismatch
[b8fa5a84] ethereumjs/eips/eip3860/initcode_limit_create_opcode/009/49121_bytes/shanghai: payload status mismatch
[b8fa5a84] ethereumjs/eips/eip3860/initcode_limit_create_opcode/004/empty/shanghai: payload status mismatch
[b8fa5a84] ethereumjs/eips/eip3860/initcode_limit_create_opcode/005/single_byte/shanghai: payload status mismatch
[b8fa5a84] ethereumjs/withdrawals/withdrawals/use_value_in_tx/001/shanghai: payload status mismatch
[b8fa5a84] ethereumjs/eips/eip3860/initcode_limit_create2_opcode/006/32_bytes/shanghai: payload status mismatch
[b8fa5a84] ethereumjs/eips/eip3860/initcode_limit_create2_opcode/007/33_bytes/shanghai: payload status mismatch
[b8fa5a84] ethereumjs/eips/eip3860/initcode_limit_create_opcode/001/max_size_ones/shanghai: payload status mismatch
[b8fa5a84] ethereumjs/eips/eip3860/initcode_limit_create_opcode/000/max_size_zeros/shanghai: payload status mismatch
[b8fa5a84] ethereumjs/eips/eip3860/initcode_limit_create2_opcode/004/empty/shanghai: payload status mismatch
[b8fa5a84] ethereumjs/eips/eip3860/initcode_limit_create2_opcode/009/49121_bytes/shanghai: payload status mismatch
[b8fa5a84] ethereumjs/eips/eip3860/initcode_limit_create2_opcode/003/over_limit_ones/shanghai: payload status mismatch
[b8fa5a84] ethereumjs/eips/eip3860/initcode_limit_create_opcode/002/over_limit_zeros/shanghai: payload status mismatch
[b8fa5a84] ethereumjs/eips/eip3860/initcode_limit_create_opcode/007/33_bytes/shanghai: payload status mismatch
[b8fa5a84] ethereumjs/eips/eip3860/initcode_limit_create_opcode/003/over_limit_ones/shanghai: payload status mismatch
[b8fa5a84] ethereumjs/eips/eip3860/initcode_limit_create_opcode/006/32_bytes/shanghai: payload status mismatch
[b8fa5a84] ethereumjs/eips/eip3860/initcode_limit_create_opcode/008/49120_bytes/shanghai: payload status mismatch
[b8fa5a84] ethereumjs/eips/eip3860/initcode_limit_create2_opcode/005/single_byte/shanghai: payload status mismatch
[b8fa5a84] ethereumjs/eips/eip3860/initcode_limit_create2_opcode/008/49120_bytes/shanghai: payload status mismatch
[b8fa5a84] ethereumjs/eips/eip3860/initcode_limit_create2_opcode/002/over_limit_zeros/shanghai: payload status mismatch
INFO[04-06|00:44:58] API: test ended                          suite=0 test=1 pass=true
INFO[04-06|00:44:58] API: suite ended                         suite=0
INFO[04-06|00:44:59] simulation ethereum/pyspec finished      suites=1 tests=94 failed=22

Will continue tomorrow.

[jochem-brouwer]

Current failing:

go run hive.go --docker.output --client ethereumjs --sim ethereum/pyspec --sim.limit "pyspec/eips/eip3860/initcode_limit_create2_opcode/002/over_limit_zeros/shanghai"

Fix by 7cc4c0f ensures the genesis has the correct state root. However, it now complains that "block 0 is not executed", I think this is because these trie nodes of genesis are not written to disk? So VM trie has no access to it (parent state root (e.g. genesis state root) unknown to VM). (Also this is somewhat a big problem, it thus seems we cannot init a custom chain client if we provide genesis state..?)

[jochem-brouwer]

At commit 77287ba:

[b7413058] failing tests:
[b7413058] ethereumjs/withdrawals/withdrawals/use_value_in_contract/000/shanghai: payload status mismatch
[b7413058] ethereumjs/withdrawals/withdrawals/use_value_in_tx/001/shanghai: payload status mismatch

Only 2 failing.

[acolytec3]

LTGM - I ran the hive tests and they all passed as well.