Adapt testeth to be used with evmlab

[gumb0]

This summarizes the requirements for a tool that could be used to conveniently run the state test files and possibly be used by fuzzing infrastructure as mentioned by @holiman #4613 (comment)

It looks like testeth currently is not too far from these requirements.

Summary of requirements

Input

• --statetest <path>
• --json - boolean flag, output json output for each opcode or not (it's useful to disable json when benchmarking)
• --nomemory - disable showing the full memory output for each op

Output

{"pc":0,"op":96,"gas":"0x2540be400","gasCost":"0x3","memory":"0x","memSize":0,"stack":[],"depth":1,"error":null,"opName":"PUSH1"}
{"pc":2,"op":0,"gas":"0x2540be3fd","gasCost":"0x0","memory":"0x","memSize":0,"stack":["0x40"],"depth":1,"error":null,"opName":"STOP"}
{"output":"","gasUsed":"0x3","time":141485}

It should output a json object for each operation. Example:

{"pc":0,"op":96,"gas":"0x2540be400","gasCost":"0x3","memory":"0x","memSize":0,"stack":[],"depth":1,"error":null,"opName":"PUSH1"}

Required: pc, op, gas, stack, depth
Optional: opName, gasCost, error

The stack, memory and memSize are the values before execution of the op.

At the end of execution, some summarical info is good, e.g.

{"output":"","gasUsed":"0x3","time":141485}

Another thing mentioned by Martin #4613 (comment) :

One important aspect is that the execution should not hold on to the data during the entire execution, and spit it out as a json array. Instead, it should output jsonl, one line per op.

testeth status

1. --testfile parameter -- should work after fix testeth -t <TestSuite> -- --singletest <path> <testname> running a test from file #5750 is finished. However -t GeneralStateTests probably will still be required, for now it's easier for us to leave it.

So the command line to run a test would look like

test/testeth -t GeneralStateTests -- --testfile /home/andrei/dev/sar00.json --jsontrace {}

2. Disabling json output for each opcode. Omitting --jsontrace disables it.

test/testeth -t GeneralStateTests -- --singletest /home/andrei/dev/sar00.json sar00

However, it will not output any summary (output, gasUsed, stateRoot etc.) either in this case.
We need to fix this (always output json summary in --testfile mode)

3. Disabling memory output. This works with --jsontrace '{"disableMemory" : true}'

test/testeth -t GeneralStateTests -- --testfile /home/andrei/dev/sar00.json --jsontrace {"disableMemory" : true}

(there're also options to disable stack and storage output)

4. Output currently looks like this

>test/testeth -t GeneralStateTests -- --testfile /home/andrei/dev/sar00.json --jsontrace {}
Test Case "customTestSuite": 
{"depth":"0","gas":"379000","gasCost":"3","memory":[],"op":"PUSH1","pc":"0","stack":[],"storage":{"0x00":"0x03"}}
{"depth":"0","gas":"378997","gasCost":"3","op":"PUSH1","pc":"2","stack":["0x00"]}
{"depth":"0","gas":"378994","gasCost":"3","op":"SAR","pc":"4","stack":["0x00","0x00"]}
{"depth":"0","gas":"378991","gasCost":"3","op":"PUSH1","pc":"5","stack":["0x00"]}
{"depth":"0","gas":"378988","gasCost":"0","op":"SSTORE","pc":"7","stack":["0x00","0x00"]}
{"depth":"0","gas":"373988","gasCost":"0","op":"STOP","pc":"8","stack":[],"storage":{}}
{"stateRoot": "a4b37b548bc12ee198ce885d83b511302d9d8f6b2ec5775618b3893c60e0dc75"}
*** No errors detected

This looks close, but

a. There are additional lines on top and at the bottom. They come from boost.test, but we could look into how to suppress them if that's important.
b. We need to add memSize field (add this in StandardTrace::operator())
c. op should have opcode number and opName the mnemonic
d. depth is off-by-one comparing to Martin's examples. This also can be adjusted in StandardTrace::operator()
e. We need to add the summary line {"output":"","gasUsed":"0x3","time":141485}
That should be added here:

aleth/test/tools/libtesteth/ImportTest.cpp

Lines 351 to 359 in 9f31c8b

	if (Options::get().jsontrace)
	{
	StandardTrace st;
	st.setShowMnemonics();
	st.setOptions(Options::get().jsontraceOptions);
	out = initialState.execute(_env, *se.get(), _tr, Permanence::Committed, st.onOp());
	cout << st.multilineTrace();
	cout << "{\"stateRoot\": \"" << initialState.rootHash().hex() << "\"}";
	}

The required data (except execution time) is returned from State::execute. Execution time is probably not so important (but it's easy to add, too)

5. testeth accumulates all the traces of the execution, then in the end outputs all of them (in the same code that I linked above). This can be fixed by modifying StandartTrace class to have an option to output immediately.

@holiman To summarize, there's somewhat quirky CLI to run it, there are some additional non-json lines in the output, the rest will be fixed to match the requirements of evmlab.

cc @winsvega

TODO

• Add memSize field
• Add opName field and change op to have number
• Make depth start with 1
• Add summary line at least with output and gasUsed
• Output summary line in --testfile mode even without --jsontrace
• Output one line per op immediately

[gumb0]

This looks done, output currently looks like this

> test/testeth -t GeneralStateTests -- --testfile /home/andrei/dev/statetest1.json   --jsontrace {}
Test Case "customTestSuite": 
{"depth":"1","gas":"2956484","gasCost":"1","memSize":0,"memory":[],"op":91,"opName":"JUMPDEST","pc":"0","stack":[],"storage":{}}
{"depth":"1","gas":"2956483","gasCost":"2","memSize":0,"memory":[],"op":61,"opName":"RETURNDATASIZE","pc":"1","stack":[]}
{"depth":"1","gas":"2956481","gasCost":"2","memSize":0,"memory":[],"op":48,"opName":"ADDRESS","pc":"2","stack":["0x00"]}
{"depth":"1","gas":"2956479","gasCost":"3","memSize":0,"memory":[],"op":172,"opName":"PUSHC","pc":"3","stack":["0x00","0xc94f5374fce5edbc8e2a8697c15331677e6ebf0b"]}
{"depth":"1","gas":"2956476","gasCost":"3","memSize":0,"memory":[],"op":172,"opName":"PUSHC","pc":"30","stack":["0x00","0xc94f5374fce5edbc8e2a8697c15331677e6ebf0b","0xc940b5f2046740058558468f238b85db7f6bbe3f3d51e92a3e32"]}
{"depth":"1","gas":"2956473","gasCost":"3","memSize":0,"memory":[],"op":83,"opName":"MSTORE8","pc":"40","stack":["0x00","0xc94f5374fce5edbc8e2a8697c15331677e6ebf0b","0xc940b5f2046740058558468f238b85db7f6bbe3f3d51e92a3e32","0xb7f7c4147541c695f3"]}
{"output": "", "gasUsed": "3000000", "error": "OutOfGas"}
{"stateRoot": "a2b3391f7a85bf1ad08dc541a1b99da3c591c156351391f26ec88c557ff12134"}

*** No errors detected

with disabled memory dump:

> test/testeth -t GeneralStateTests -- --testfile /home/andrei/dev/statetest1.json   --jsontrace '{"disableMemory": true}'
Test Case "customTestSuite": 
{"depth":"1","gas":"2956484","gasCost":"1","memSize":0,"op":91,"opName":"JUMPDEST","pc":"0","stack":[],"storage":{}}
{"depth":"1","gas":"2956483","gasCost":"2","memSize":0,"op":61,"opName":"RETURNDATASIZE","pc":"1","stack":[]}
{"depth":"1","gas":"2956481","gasCost":"2","memSize":0,"op":48,"opName":"ADDRESS","pc":"2","stack":["0x00"]}
{"depth":"1","gas":"2956479","gasCost":"3","memSize":0,"op":172,"opName":"PUSHC","pc":"3","stack":["0x00","0xc94f5374fce5edbc8e2a8697c15331677e6ebf0b"]}
{"depth":"1","gas":"2956476","gasCost":"3","memSize":0,"op":172,"opName":"PUSHC","pc":"30","stack":["0x00","0xc94f5374fce5edbc8e2a8697c15331677e6ebf0b","0xc940b5f2046740058558468f238b85db7f6bbe3f3d51e92a3e32"]}
{"depth":"1","gas":"2956473","gasCost":"3","memSize":0,"op":83,"opName":"MSTORE8","pc":"40","stack":["0x00","0xc94f5374fce5edbc8e2a8697c15331677e6ebf0b","0xc940b5f2046740058558468f238b85db7f6bbe3f3d51e92a3e32","0xb7f7c4147541c695f3"]}
{"output": "", "gasUsed": "3000000", "error": "OutOfGas"}
{"stateRoot": "a2b3391f7a85bf1ad08dc541a1b99da3c591c156351391f26ec88c557ff12134"}

*** No errors detected

Note 1. gasUsed above shows that all of the gas was consumed, as execution finished with an error, this looks correct to me. geth's evm tool outputs some other value (gas left before error?)

Note 2. storage, as before, (unless completely disabled) is dumped only for opcodes changing the storage and after the execution frame changes.

cc @holiman would be great if this is useful

[holiman]

Problem, when I run it (and also in your dump above)

=== RUN   TestVMsOutput
geth: {"pc":0,"op":91,"gas":"0x2d1cc4","gasCost":"0x1","memory":"0x","memSize":0,"stack":[],"depth":1,"refund":0,"opName":"JUMPDEST","error":""}
geth: {"pc":1,"op":61,"gas":"0x2d1cc3","gasCost":"0x2","memory":"0x","memSize":0,"stack":[],"depth":1,"refund":0,"opName":"RETURNDATASIZE","error":""}
geth: {"pc":2,"op":48,"gas":"0x2d1cc1","gasCost":"0x2","memory":"0x","memSize":0,"stack":["0x0"],"depth":1,"refund":0,"opName":"ADDRESS","error":""}
geth: {"pc":3,"op":121,"gas":"0x2d1cbf","gasCost":"0x3","memory":"0x","memSize":0,"stack":["0x0","0xc94f5374fce5edbc8e2a8697c15331677e6ebf0b"],"depth":1,"refund":0,"opName":"PUSH26","error":""}
geth: {"pc":30,"op":104,"gas":"0x2d1cbc","gasCost":"0x3","memory":"0x","memSize":0,"stack":["0x0","0xc94f5374fce5edbc8e2a8697c15331677e6ebf0b","0xc940b5f2046740058558468f238b85db7f6bbe3f3d51e92a3e32"],"depth":1,"refund":0,"opName":"PUSH9","error":""}
geth: {"pc":40,"op":83,"gas":"0x2d1cb9","gasCost":"0x3","memory":"0x","memSize":0,"stack":["0x0","0xc94f5374fce5edbc8e2a8697c15331677e6ebf0b","0xc940b5f2046740058558468f238b85db7f6bbe3f3d51e92a3e32","0xb7f7c4147541c695f3"],"depth":1,"refund":0,"opName":"MSTORE8","error":"gas uint64 overflow"}
geth: {"output":"","gasUsed":"0x2d1cc4","time":233624,"error":"gas uint64 overflow"}
geth: {"stateRoot": "a2b3391f7a85bf1ad08dc541a1b99da3c591c156351391f26ec88c557ff12134"}
aleth: Test Case "customTestSuite": 
aleth err: invalid character 'T' looking for beginning of value, line
	Test Case "customTestSuite": 
aleth: {"depth":1,"gas":"2956484","gasCost":"1","memSize":0,"op":91,"opName":"JUMPDEST","pc":0,"stack":[],"storage":{}}
aleth: {"depth":1,"gas":"2956483","gasCost":"2","memSize":0,"op":61,"opName":"RETURNDATASIZE","pc":1,"stack":[]}
aleth: {"depth":1,"gas":"2956481","gasCost":"2","memSize":0,"op":48,"opName":"ADDRESS","pc":2,"stack":["0x00"]}
aleth: {"depth":1,"gas":"2956479","gasCost":"3","memSize":0,"op":172,"opName":"PUSHC","pc":3,"stack":["0x00","0xc94f5374fce5edbc8e2a8697c15331677e6ebf0b"]}
aleth: {"depth":1,"gas":"2956476","gasCost":"3","memSize":0,"op":172,"opName":"PUSHC","pc":30,"stack":["0x00","0xc94f5374fce5edbc8e2a8697c15331677e6ebf0b","0xc940b5f2046740058558468f238b85db7f6bbe3f3d51e92a3e32"]}
aleth: {"depth":1,"gas":"2956473","gasCost":"3","memSize":0,"op":83,"opName":"MSTORE8","pc":40,"stack":["0x00","0xc94f5374fce5edbc8e2a8697c15331677e6ebf0b","0xc940b5f2046740058558468f238b85db7f6bbe3f3d51e92a3e32","0xb7f7c4147541c695f3"]}
aleth: {"output": "", "gasUsed": "3000000", "error": "OutOfGas"}
aleth: {"stateRoot": "a2b3391f7a85bf1ad08dc541a1b99da3c591c156351391f26ec88c557ff12134"}
diff: 

Geth outputs PUSH26 and PUSH9 respectively, whereas aleth/testeth outputs 172:PUSHC. The textual representation isn't a problem , but showing 172 in lieu of the actual opcodes is problematic. How do I disable that?

[holiman]

Hmm, I seem to recall that there's some special build-flag to remove those, right?