[test] Extraction of 292 tests from SolidityEndToEndTest.cpp

[aarlt]

• verified correctness of extracted tests with scripts/end-to-end/verify-testcases.py

[aarlt]

./verify-testcases.py
addmod_mulmod_zero
    kind='call' parameter='1212121212121212121212121212120000000012' input='' output='' value='' result='4'
    kind='call' parameter='1212121212121212121212121212120000000012' input='' output='' value='' result='2'
                                                                                                         |
292 test-cases -  1  mismatche(s)

The cause of the mismatch was a typo within the original boost test-case:

ABI_CHECK(callContractFunction("f(uint)", 0), encodeArgs());
ABI_CHECK(callContractFunction("g(uint)", 0), encodeArgs());
ABI_CHECK(callContractFunction("h()"), encodeArgs(2));

signatures need to be f(uint256) and g(uint256)

[aarlt]

@chriseth This PR is based on the verification of the original test traces (created with./soltest --log_level=test_suite -t SolidityEndToEndTest/ -- --no-smt --show-messages) vs. the traces created by the extracted tests (created with ./soltest --log_level=test_suite -t semanticTests/extracted -- --no-smt --show-messages). For the comparation of these traces scripts/end-to-end/verify-testcases.py was used.

I also added the scripts/end-to-end/remove-testcases.py script here. From my point of view someone can just add newly extracted files (manually or automated) to semanticTests/extracted, create the traces, run the verifier and if everything was ok, execute scripts/end-to-end/remove-testcases.py > SolidityEndToEndTest.cpp.

[ekpyron]

@aarlt Could you add a command line option like -i to remove-testcases.py that puts it in "interactive mode"? I.e. for each test case it outputs the whole BOOST_AUTO_TEST_CASE(...) {...} part it is going to remove together with the corresponding extracted file and waits for user input to continue.
That way I could quickly step through the 292 test cases and confirm visually for a second or so that it looks ok. That plus looking at the traces I would consider enough to approve this.

[ekpyron]

@aarlt Also what about the plan to also add the metadata to the traces and to compare it (excluding the sources)?

[aarlt]

@aarlt Could you add a command line option like -i to remove-testcases.py that puts it in "interactive mode"? I.e. for each test case it outputs the whole BOOST_AUTO_TEST_CASE(...) {...} part it is going to remove together with the corresponding extracted file and waits for user input to continue.
That way I could quickly step through the 292 test cases and confirm visually for a second or so that it looks ok. That plus looking at the traces I would consider enough to approve this.

Thats a really nice idea!

[aarlt]

@aarlt Also what about the plan to also add the metadata to the traces and to compare it (excluding the sources)?

I will do that today. I will create a new PR for this.

[chriseth]

With the single step thing, this looks good, I would say. Can you do it in a way that it displays a diff somehow? At best with "inline diff", ignoring whitespace and marking intra-line changes bold

[aarlt]

@ekpyron @chriseth I added a simple interactive mode to view the extracted test-cases. I was not really able to find a nice way to visualise with diff. I hope that the simple visualisation I've implemented here helps a bit.

usage: ./remove-testcases.py [-i] [-f <full path to SolidityEndToEndTest.cpp>]

You should point with -f to a location of SolidityEndToEndTest.cpp file of the develop branch.

[ekpyron]

Two initial comments for the scripts - I'll have a look at an interactive run.

[ekpyron]

Can you avoid requiring a return confirmation for the (y/n) answer? So that hitting a y or n key once suffices without further confirmation?

[aarlt]

@chriseth @ekpyron I added a simple bash script create_traces.sh that will create the needed trace-files for verify-testcases.py. verify-testcases.py will compare the metadata, all create and call messages. Basically it just compares the traces of the original end-to-end tests (SolidityEndToEndTest.cpp) with the traces that where generated by the extracted tests. Just execute ./verify-testcases.py.

To be able to review the extracted test-cases the remove-testcases.py script can be used in interactive mode. You will need to point to a SolidityEndToEndTest.cpp file where the tests where not yet extracted, e.g. use the current develop branch. If you use create_traces.sh, you can use the already checked out develop branch to point to SolidityEndToEndTest.cpp with ./remove-testcases.py -i -f solidity-develop/test/libsolidity/SolidityEndToEndTest.cpp.

The diff-tool used by the remove-testcases.py script can be defined by setting the DIFF environment variable.

[chriseth]

Scripts look ok, will take a look at the tests now.