Understanding and debugging Nimbus EVM JSON tests

Introduction

The Nimbus JSON tests are taken from the official Ethereum test suite. To find out more about the contents of this test suite, please refer to the official documentation.

You can run all tests with nimble test. Based on our config, Nimble will write binaries to build/ - you can do this manually also, as in the following examples:

Run example:

mkdir -p build
nim c -r -o:build/decompile_smart_contract examples/decompile_smart_contract.nim

General State Tests

Run Ethereum JSON-based general state tests:

mkdir -p build
nim c -r -o:build/test_generalstate_json tests/test_generalstate_json.nim

You might want to run the tests with -d:release flag as some tests might end up with a stack overflow in debug mode.

Build and run an individual test (with tracing enabled):

nim c -r -d:chronicles_log_level=TRACE -o:build/test_generalstate_json tests/test_generalstate_json.nim stAttackTest/ContractCreationSpam.json

Individual tests are default only tested on Frontier fork. You can change the fork with --fork:, e.g.:

nim c -d:release -r -o:build/test_generalstate_json tests/test_generalstate_json.nim stCallCreateCallCodeTest/Call1024BalanceTooLow.json --fork:Byzantium

Subtest for this individual test can be selected using --index:number, and the number start from 1.

nim c -d:release -r -o:build/test_generalstate_json tests/test_generalstate_json.nim stSStoreTest/InitCollision.json --fork:Byzantium --index:1

Every time you run this individual test, it will dump debugging data/VM trace in json formatted file. If you only need short dump instead of long dump, you can use --trace:on/off or --trace:true/false.

Block Chain Tests

The same CLI are used both by General State Tests and Block Chain Tests. They differ in the input path and doesn't accept fork argument.

nim c -d:release -r tests/test_blockchain_json.nim GeneralStateTests/stSStoreTest/InitCollision.json --index:1

Addition input path prefix are:

• GeneralStateTests
• InvalidBlocks
• TransitionTests
• ValidBlocks

VMTests

JSON structure

A JSON EVM test has the following structure (example is add0.json).

{
    "add0" : {
        "_info" : {
            "comment" : "",
            "filledwith" : "cpp-1.3.0+commit.6e0ce939.Linux.g++",
            "lllcversion" : "Version: 0.4.18-develop.2017.9.25+commit.a72237f2.Linux.g++",
            "source" : "src/VMTestsFiller/vmArithmeticTest/add0Filler.json",
            "sourceHash" : "dcc7fc8aebdc2d7334440cfe6c63172941b4164c1ba8c32897318ca0cdfb7a1c"
        },
        "callcreates" : [
        ],
        "env" : {
            "currentCoinbase" : "0x2adc25665018aa1fe0e6bc666dac8fc2697ff9ba",
            "currentDifficulty" : "0x0100",
            "currentGasLimit" : "0x0f4240",
            "currentNumber" : "0x00",
            "currentTimestamp" : "0x01"
        },
        "exec" : {
            "address" : "0x0f572e5295c57f15886f9b263e2f6d2d6c7b5ec6",
            "caller" : "0xcd1722f2947def4cf144679da39c4c32bdc35681",
            "code" : "0x7fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff7fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff01600055",
            "data" : "0x",
            "gas" : "0x0186a0",
            "gasPrice" : "0x5af3107a4000",
            "origin" : "0xcd1722f2947def4cf144679da39c4c32bdc35681",
            "value" : "0x0de0b6b3a7640000"
        },
        "gas" : "0x013874",
        "logs" : "0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347",
        "out" : "0x",
        "post" : {
            "0x0f572e5295c57f15886f9b263e2f6d2d6c7b5ec6" : {
                "balance" : "0x0de0b6b3a7640000",
                "code" : "0x7fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff7fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff01600055",
                "nonce" : "0x00",
                "storage" : {
                    "0x00" : "0xfffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffe"
                }
            }
        },
        "pre" : {
            "0x0f572e5295c57f15886f9b263e2f6d2d6c7b5ec6" : {
                "balance" : "0x0de0b6b3a7640000",
                "code" : "0x7fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff7fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff01600055",
                "nonce" : "0x00",
                "storage" : {
                }
            }
        }
    }
}

Important: As of July 2018, VMTests always use Homestead fork for opcode implementations (notably CALL) and gas prices. We should not rely on currentNumber.

The important fields are:

• exec section:

  • code: the code being executed in isolation. (In production this is part of a transaction).
  • gas: usually 0x0186a0 (100000) the starting gas

• gas field: the remaining gas after code execution. Here 0x013874 (79988)

• pre section: the state before execution of the code by the VM

  • storage: if the account already stores data that we can retrieve with SLOAD or overwrite/reset with SSTORE

• post section: the state after execution of the code by the VM. If there is no post section, the code is supposed to throw an EVM exception. After an EVM exception, gas is consumed but state is reverted.

  Implementation-wise computation.isError returns true and an error message is available in computation.error field (implementation as of July 2018)

Decompiling bytecode.

You can decompile EVM bytecode using Etherscan: https://etherscan.io/opcode-tool Pasting 0x6003600202600055 gives:

[1] PUSH1 0x03 
[3] PUSH1 0x02 
[4] MUL 
[6] PUSH1 0x00 
[7] SSTORE 

Alternatively you can use the following:

import ../nimbus/vm/code_stream, strformat

var c = newCodeStreamFromUnescaped("0x6003600202600055")

let opcodes = c.decompile()
for op in opcodes:
  echo &"[{op[0]}]\t{op[1]}\t{op[2]}"

# [1]     PUSH1   0x03
# [3]     PUSH1   0x02
# [4]     MUL
# [6]     PUSH1   0x00
# [7]     SSTORE
# [-1]    STOP

The number in bracket refers to the position after reading the opcode and its arguments (i.e. the value of the program counter).

add0 (0x7fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff7fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff01600055) decompiles to:

[32] PUSH32 0xffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff 
[65] PUSH32 0xffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff 
[66] ADD 
[68] PUSH1 0x00 
[69] SSTORE

Debugging output

Nimbus outputs several tiers of debug information using the Chronicles library.

These are roughly split into several layers:

• Trace
• Debug
• Info
• Notice
• Warn
• Error
• Fatal

In code, these look like this:

debug "Some debugging stuff", currentValue = value

Chronicles allows several levels of control over output, and these can be controlled by compiler flags when building Nimbus. The Chronicles repo goes into depth about how these flags operate.

Selecting output

Levels

When compiling using Nim with debug, Chronicles defaults to displaying debug and above messages, however this can be overridden by passing -d:chronicles_log_level= with one of the above levels. For example, you can compile using Nim's debug, but show only info level messages with -d:chronicles_log_level=INFO.

Colours

Some terminals don't support colour output and in this case you will see lots of extra characters that makes it difficult to read messages. To turn colouring off, you can use: -d:chronicles_sinks=textlines[nocolors, stdout]

Testing in other clients

py-evm

• run a single JSON test with tracing enabled (by the logging level):

pytest -o log_cli=true --log-cli-level=NOTSET -k fixtures/GeneralStateTests/stAttackTest/ContractCreationSpam.json tests/json-fixtures/test_state.py --fork Homestead

Installation hints:

• use a virtualenv
• pip install -e .[dev] might fail with a dependency conflict (which you can see later on with pip check). You work around that by manually downgrading the offending packages, like this:

pip install pluggy==0.7.1
pip install idna==2.7
# as of 2018-12-06, you also need:
pip install -e ./trinity-external-plugins/examples/peer_count_reporter

• upstream runs tests using tox, so start from there to get to the relevant pytest arguments:

tox -l
tox -e py36-native-state-homestead
pytest tests/json-fixtures/test_state.py --fork Homestead

go-ethereum

./build/env.sh go test -v -run TestState/stAttackTest/ContractCreationSpam.json ./tests