Extract semantic tests Pt. 1 #5736

erak · 2019-01-04T23:58:03Z

Description

Part of #4223.
Based on #4245.

This PR adds support for running semantic (end-to-end) tests via isoltest. It enables:

calling functions with one argument value and one return value
Argument / return type encodings: uint, int (unpadded)
Test cases like:

contract test {
    function f(uint a) public returns(uint d) { return a ** 0; }
}
// ----
// f(uint256): 1 # A comment
// -> 1

A test that fails, creates an output of this form:

semanticTests/empty_contract.sol: FAIL
  Contract:
    contract test { }

  Expected result:
    i_am_not_there() # This fails.
    -> 1
  Obtained result:
    i_am_not_there() # This fails.
    REVERT

Not supported:

Argument / return type encodings: string, bool
Multiple arguments, return values:

// ----
// f(uint256, uint256): 1, 1
// -> bool

Todo

Extend documentation (also inline)

Follow-up PRs

Let tests fail that don't specify expectations (Exception vs. fail?)
If no-ipc is specified, at least compile tests

codecov · 2019-01-07T14:03:12Z

Codecov Report

Merging #5736 into develop will decrease coverage by 0.22%.
The diff coverage is 57.97%.

@@             Coverage Diff             @@
##           develop    #5736      +/-   ##
===========================================
- Coverage    88.35%   88.13%   -0.23%     
===========================================
  Files          348      352       +4     
  Lines        33431    33600     +169     
  Branches      4005     4045      +40     
===========================================
+ Hits         29538    29612      +74     
- Misses        2535     2602      +67     
- Partials      1358     1386      +28

Flag	Coverage Δ
#all	`88.13% <57.97%> (-0.23%)`	⬇️
#syntax	`28.16% <2.38%> (-0.17%)`	⬇️

chriseth · 2019-01-07T16:31:31Z

test/InteractiveTests.h

-	{"JSON AST",            "libsolidity", "ASTJSON",             false, false, &ASTJSONTest::create},
-	{"SMT Checker",         "libsolidity", "smtCheckerTests",     true,  false, &SyntaxTest::create},
-	{"SMT Checker JSON",    "libsolidity", "smtCheckerTestsJSON", true,  false, &SMTCheckerTest::create}
+//	{"Yul Optimizer",       "libyul",      "yulOptimizerTests",   false, false, &yul::test::YulOptimizerTest::create},


Should not stay in the final PR.

chriseth · 2019-01-07T16:32:20Z

test/libsolidity/SemanticTest.cpp

+		if(_actualResults && _formatted)
+			_stream << endl;
+
+//		string result;


should not stay in the final PR

chriseth · 2019-01-07T16:33:11Z

test/libsolidity/ExpectationParser.h

+namespace test
+{
+
+class ExpectationParser


Please document.

chriseth · 2019-01-07T16:33:21Z

test/libsolidity/ExpectationParser.h

+class ExpectationParser
+{
+public:
+	struct FunctionCallResult {


{ to the next line

chriseth · 2019-01-07T16:34:23Z

test/libsolidity/ExpectationParser.cpp

+	bytes byteRange(it, _bytes.end());
+	stringstream resultStream;
+	// TODO: Convert from compact big endian if padded
+	resultStream << fromBigEndian<u256>(byteRange);


could directly take _bytes

chriseth · 2019-01-07T16:40:22Z

test/libsolidity/ExpectationParser.cpp

+	}
+}
+
+string ExpectationParser::bytesToString(bytes const& _bytes)


There are decimal integers involved here, so I think the name should be different :)

stackenbotten · 2019-01-07T17:00:34Z

There was an error when running test_check_style for commit 51df66a5fe2a65c8ec7fdf9d8ef7b53cfec2a334:

Error: Trailing whitespace found:
 test/libsolidity/SemanticTest.cpp:59: {

Please check that your changes are working as intended.

chriseth · 2019-01-21T11:37:30Z

Changelog.md

@@ -6,6 +6,7 @@ Language Features:

 Compiler Features:
 * Control Flow Graph: Warn about unreachable code.
+ * Tests: Add infrastructure to ``isoltest`` for running semantic tests.


I think it would be better to move that to "build system" or rename "Build system" to "infrastructure" or something like that. At least it is nothing that concerns the user.

chriseth · 2019-01-21T11:39:39Z

test/libsolidity/semanticTests/exp_operator.sol

+// f(uint256): 1
+// -> 2
+// g(uint256, uint256): 1, 1
+// REVERT # Argument encoding is not properly implemented, yet.


Will the comment stay after a call to "update"?

It might be easier to add the comment to the function call and not the the expectation.

Hm, in general, it might make sense to allow newlines (which will be preserved during update) in the list of arguments. Is that supported?

Basically this would mean that function call is everything up to a line that starts with REVERT or ->.

New lines for function calls are not supported as of now. To clarify, your proposal is to have the call and the expectations written down like this:

// f(uint256) // 1 // -> 2 // g(uint256, uint256) // 1, 1 // REVERT # Argument encoding is not properly implemented, yet.

Is that correct? I'd also slightly tend towards that solution as it would have the benefit to support not only Ether that is send (currently: f(uint256): 1 [etherValue]), but also argument ranges that I'd like to implement at some point:

// f(uint256, uint256): [etherValue] // [1...10], 1 // -> 2

In my opinion, this would be consistent as everything that is send with the function call (arguments and ether) is encoded in brackets. But maybe that's a bit confusing and we should keep the ether value in brackets only. Supporting argument ranges could simply work like this: 1..10, 1.

It might be easier to add the comment to the function call and not the the expectation.

The comment is preserved on both lines already. So I guess it's fine for now. Would need to make sure that this works if we're going to add another extra line for the arguments.

Ah, I was suggesting a relaxed handling of newline characters: I would assume that sometimes, the arguments can be rather large and the argument list long, so you would want to be able to insert newlines at convenient places.

After reading the comment again, I also got your intention.

chriseth · 2019-01-22T10:00:23Z

I'm sorry, I think it was also my fault that this PR got so big. Are there reasonable parts to extract into other PRs?

erak · 2019-01-22T12:14:41Z

Yea, that's the reason why I left out some aspects in the first place. One the other hand it's a good opportunity to split out the expectation parser and properly test it. The other part will then be the actual execution environment.

erak · 2019-01-24T10:02:08Z

Closing this now in favor of #5860 and following.

erak changed the title ~~Extract semantic tests Pt. 1~~ [WIP] Extract semantic tests Pt. 1 Jan 5, 2019

erak force-pushed the semantic-tests branch from cc157b6 to c33ce83 Compare January 7, 2019 13:09

chriseth reviewed Jan 7, 2019

View reviewed changes

test/libsolidity/ExpectationParser.h

namespace test

{

class ExpectationParser

Copy link

Contributor

chriseth Jan 7, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please document.

chriseth reviewed Jan 7, 2019

View reviewed changes

erak force-pushed the semantic-tests branch 2 times, most recently from e93237d to 34a8cfc Compare January 19, 2019 22:06

Adds skeleton of semantic tests to isoltest.

bde8a7e

erak force-pushed the semantic-tests branch from 34a8cfc to 36bfc2a Compare January 19, 2019 23:14

erak added 3 commits January 20, 2019 00:45

Moves batch of end-to-end tests to isoltest.

e71541f

Runs semantic tests /w support for numeric types.

3dab0e9

Introduces scanner to semantic test parser.

ae23db6

erak force-pushed the semantic-tests branch 2 times, most recently from 49ee8a4 to b7558a4 Compare January 20, 2019 23:08

erak changed the title ~~[WIP] Extract semantic tests Pt. 1~~ Extract semantic tests Pt. 1 Jan 20, 2019

erak added 3 commits January 21, 2019 11:29

Cleans up semantic tests implementation. (to be squashed)

b345b10

Passes IPC path to semantic test properly. (to be squashed)

160a9ec

Enables updating expectations of semantic tests. (to be squashed)

5897f89

erak force-pushed the semantic-tests branch from b113dbc to 5897f89 Compare January 21, 2019 10:30

chriseth reviewed Jan 21, 2019

View reviewed changes

Preserves format of expectations in semantic tests.

1fb08c0

erak force-pushed the semantic-tests branch from b429a76 to 1fb08c0 Compare January 21, 2019 23:31

erak closed this Jan 24, 2019

erak deleted the semantic-tests branch August 14, 2019 09:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Extract semantic tests Pt. 1 #5736

Extract semantic tests Pt. 1 #5736

erak commented Jan 4, 2019 •

edited

Loading

codecov bot commented Jan 7, 2019 •

edited

Loading

chriseth Jan 7, 2019

chriseth Jan 7, 2019

chriseth Jan 7, 2019

chriseth Jan 7, 2019

chriseth Jan 7, 2019

chriseth Jan 7, 2019

stackenbotten commented Jan 7, 2019

chriseth Jan 21, 2019

chriseth Jan 21, 2019

chriseth Jan 21, 2019

chriseth Jan 21, 2019

chriseth Jan 21, 2019

erak Jan 21, 2019

erak Jan 21, 2019 •

edited

Loading

chriseth Jan 22, 2019

erak Jan 22, 2019

chriseth commented Jan 22, 2019

erak commented Jan 22, 2019

erak commented Jan 24, 2019

Extract semantic tests Pt. 1 #5736

Extract semantic tests Pt. 1 #5736

Conversation

erak commented Jan 4, 2019 • edited Loading

Description

Todo

Follow-up PRs

codecov bot commented Jan 7, 2019 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stackenbotten commented Jan 7, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erak Jan 21, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chriseth commented Jan 22, 2019

erak commented Jan 22, 2019

erak commented Jan 24, 2019

erak commented Jan 4, 2019 •

edited

Loading

codecov bot commented Jan 7, 2019 •

edited

Loading

erak Jan 21, 2019 •

edited

Loading