[WIP] Extract solidity end-to-end tests

[aarlt]

To generate the given solidity files a generator was used that can be found here https://github.com/aarlt/solidity/tree/62c1515bc45c4bcc587dfd52458ca783c5ff9cd1/test/tools/end2end-extractor.

The generator is checking for the used features per test-case. Currently, the generator detects that 180 test-cases are not extractable. Tests that are not extractable use
m_evmHost, m_output, m_contractAddress, m_compiler, try to send value or need to pass arguments to the constructor. Tests using testContractAgainstCppOnRange, testContractAgainstCpp are excluded from the extraction, because these kind of tests are tested against C++ functions and are better readable compared to resulting semantic tests. Tests that call
compileAndRun multiple times while defining different source-codes within a single test will also get excluded for now.

However, 364 test-cases seem to be extractable. As for now I was not able to write a generic encoder that allows the generation of all these test-cases without any manual change.

• extraction of simple test-cases - no changes to the generated files are needed. 259 test-cases are simple.
• extraction of test-cases that where checking for exceptions - FAILURE need to be added to the test-cases either manually or with the help of isoltest.
• complex test-cases need to be manually rewritten. These test-cases use complex encoding and the current encoder within the generator is not able to deal with ambiguity.
• run a solidity formatter to format the resulting tests nicely.
• removing extracted tests from solidity/test/libsolidity/SolidityEndToEndTest.cpp

As for now 259 / 364 test-cases where extracted.

@chriseth I'm still not sure whether this will be better reviewable :-(

Fixes #8197. Old PR #8284.

[chriseth]

Can you add the tool to the repository?

[chriseth]

As I said, I do not want to review the result, I want to review the generator. If that is fine, spot-checks to the result should be all that is needed.

[chriseth]

Also please do not use numbers to abbreviate actual words :)

[chriseth]

Run a solidity formatter: Maybe it's fine to just remove the leading tabs.

[aarlt]

Can you add the tool to the repository?

Sure. I just thought that it may not be very useful once we extracted all the extractable tests. However, I will add the generator to this PR.

As I said, I do not want to review the result, I want to review the generator. If that is fine, spot-checks to the result should be all that is needed.

Yes, I agree. That would be perfect.

Also please do not use numbers to abbreviate actual words :)

That's really a pity :D

Run a solidity formatter: Maybe it's fine to just remove the leading tabs.

Yes, I will do that.

[chriseth]

Can you explain the purpose of the individual files? Why did you need to add a fake compiler stack?

[aarlt]

Can you explain the purpose of the individual files? Why did you need to add a fake compiler stack?

• EndToEndTests.h/.cpp defines the End2EndExtractor class that is derived from ExtractorExecutionFramework. The .cpp file defines needed macros and includes SolidityEndToEndTest.cpp to "inject" the different test-cases into the End2EndExtractor. This is mainly done within the constructor of End2EndExtractor, where every BOOST_AUTO_TEST_CASE defines an ExtractionTask.

#define BOOST_AUTO_TEST_CASE(X) m_tests[#X] = ExtractionTask(#X, [this]() { \
	    prepareTest(#X, m_tests);

• ExtractionTask.h/.cpp prepareTest(..) is used to set the currently running ExtractionTask. The original boost-test-cases became lambdas that will be bound to different ExtractionTask instances in std::function<void(void)> m_task. The class is mainly used to collect "expectations" and stores information about the extractability.

• ExtractionExecutionFramework.h/.cpp defines basically the context of the running boost-test-cases. For example here where methods defined like ABI_CHECK. Some tests access some members like m_contractAddress, m_compiler, m_output or m_evmHost. These where original member variables, but I replaced them with function calls. Like e.g.

#define m_compiler extractor_m_compiler()

If those "members" where used by a specific test-case, the test-casee will be marked as not extractable.

• FakeCompilerStack.hpp and FakeEvmHost.hpp are just defining the concept of the CompilerStack or the EvmHost. These classes don't do anything. Some tests access some members of those classes. The main-reason for those classes is mainly performance. E.g. if the real compiler stack instance would have been used here, each test-case need to be actually compiled during the analysis and slows down the extraction.

[chriseth]

Why can you avoid compiling if you actually need to run? Or do you run two passes?

[aarlt]

Why can you avoid compiling if you actually need to run? Or do you run two passes?

I use one analysis pass that is mainly executing the bounded m_task function of the original BOOST_TEST_CASE. Here all information will be collected into the different ExtractionTasks. The second step is just the extraction, where the collected information are just written somewhere.

With compiling I meant the compilation of the solidity test-cases. But we already know that they are compilable. We only want to use its source-code.

[aarlt]

@chriseth I think that the cpp based test-extractor is probably a bit too complex. So I wrote more simple test-extractor in python (see https://github.com/ethereum/solidity/blob/58651e7ebd977288afa1404648395a2a4ab46fa5/test/tools/extractor.py/extractor.py). But this extractor is only able to extract the very simple test-cases. It looks like that there are 260 very simple test-cases. The python extractor is also able to preserve the comments given in the original test-cases. However, some test-cases seem to have compilation problems. I didn't investigated these problems so far.

semanticTests/end2end/pass_function_types_externally.sol: Warning: This is a pre-release compiler version, please do not use it in production.
:11:55: Error: Expected ',' but got 'external'
    function eval(function(uint256) returns (uint256) external x, uint256 a)
                                                      ^------^
Exception during test: /Users/alex/git/aarlt/solidity/libsolidity/interface/CompilerStack.cpp(531): Throw in function const std::__1::string solidity::frontend::CompilerStack::lastContractName() const
Dynamic exception type: boost::wrapexcept<solidity::langutil::CompilerError>
std::exception::what: Parsing was not successful.
[solidity::util::tag_comment*] = Parsing was not successful.

semanticTests/end2end/calling_uninitialized_function_through_array.sol: Warning: This is a pre-release compiler version, please do not use it in production.
:14:38: Error: Expected identifier but got 'internal'
        function() returns (uint256) internal[200] memory x;
                                     ^------^
Exception during test: /Users/alex/git/aarlt/solidity/libsolidity/interface/CompilerStack.cpp(531): Throw in function const std::__1::string solidity::frontend::CompilerStack::lastContractName() const
Dynamic exception type: boost::wrapexcept<solidity::langutil::CompilerError>
std::exception::what: Parsing was not successful.
[solidity::util::tag_comment*] = Parsing was not successful.

semanticTests/end2end/store_function.sol: Warning: This is a pre-release compiler version, please do not use it in production.
:10:50: Error: Expected ',' but got 'external'
    function(function(uint256) returns (uint256) external)
                                                 ^------^
Exception during test: /Users/alex/git/aarlt/solidity/libsolidity/interface/CompilerStack.cpp(531): Throw in function const std::__1::string solidity::frontend::CompilerStack::lastContractName() const
Dynamic exception type: boost::wrapexcept<solidity::langutil::CompilerError>
std::exception::what: Parsing was not successful.
[solidity::util::tag_comment*] = Parsing was not successful.

[chriseth]

Can you extract the python-based extractor and the tests it extracts into its own PR?

[aarlt]

Can you extract the python-based extractor and the tests it extracts into its own PR?

Created its own PR, see #8381

[aarlt]

Replaced by #8464 and #8518.