Semantic test infrastructure #5872

erak · 2019-01-27T17:58:03Z

Part of #4223 and based on #5860.

Adds infrastructure to the testing environment (isoltest and boost unit tests, as well), that allows to run file-based semantic tests.

Checklist

README / documentation was extended, if necessary
Change target branch after merging: Test file parser. #5860

codecov · 2019-01-28T10:26:28Z

Codecov Report

Merging #5872 into develop will decrease coverage by 0.43%.
The diff coverage is 40.29%.

@@             Coverage Diff             @@
##           develop    #5872      +/-   ##
===========================================
- Coverage    88.59%   88.16%   -0.44%     
===========================================
  Files          359      361       +2     
  Lines        34596    34774     +178     
  Branches      4089     4122      +33     
===========================================
+ Hits         30651    30657       +6     
- Misses        2581     2724     +143     
- Partials      1364     1393      +29

Flag	Coverage Δ
#all	`88.16% <40.29%> (-0.44%)`	⬇️
#syntax	`27.71% <4%> (-0.23%)`	⬇️

test/ExecutionFramework.h

test/libsolidity/SemanticTest.h

test/libsolidity/SemanticTest.cpp

ekpyron

Most of it looks good, but still, I think we should restructure it, so that only the result encoding is regenerated and the call and argument part is not touched at all (plus some minor other stuff).

test/libsolidity/SemanticTest.cpp

ekpyron · 2019-02-07T12:01:01Z

test/tools/isoltest.cpp

 	bool disableSMT = false;
 	bool formatted = true;
 	po::options_description options(
 		R"(isoltest, tool for interactively managing test contracts.
-Usage: isoltest [Options] --testpath path
+Usage: isoltest [Options] --testpath path --ipcpath ipcpath


Suggested change

Usage: isoltest [Options] --testpath path --ipcpath ipcpath

Usage: isoltest [Options] --testpath path {--ipcpath ipcpath, --no-ipc}

Or [--ipcpath ipcpath]in "optional brackets"?

Or just two lines (with or without repeating Usage: ):

Usage: isoltest [Options] --testpath path --ipcpath ipcpath Usage: isoltest [Options] --testpath path --no-ipc

I think we could follow the unix standard here and print:

Usage: isoltest [--testpath testpath] [--ipcpath ipcpath] [--no-ipc]

or

Usage: isoltest [--no-ipc] [--testpath testpath] [--ipcpath ipcpath]

Actually if we start guessing the ipcpath, then none of the options are particularly special and we could just leave it at Usage: isoltest [Options]. In any case we can remove the --testpath testpath part here - I think it's only in there as a relic from the time where it wasn't optional.

I've gone with Usage: isoltest [Options] --ipcpath ipcpath for now, since we're not guessing it, yet.

I think that "guessing" code needs to be shared with soltest, @Marenz what's your take on this?

Saw this too late. But yeah, if they share command line parameters the defaults / handling should be shared

test/tools/isoltest.cpp

test/RPCSession.cpp

test/libsolidity/SemanticTest.cpp

erak · 2019-02-15T11:44:02Z

Updated. This PR should now have all the features planned and can be reviewed again. Please note that all the test files have been updated by isoltest so this can be considered as the format spec that it follows. This formatting should now also be inline with the display-modes that the parser detects.

erak · 2019-02-15T11:45:56Z

Ah, I did not squash on purpose yet, but will do when it's ready to be merged.

ekpyron

Yeah - a lot of this needs to be touched again anyways to add more features and we can improve some details then - I'd say this is fine, once it's squashed!

test/libsolidity/SemanticTest.cpp

chriseth · 2019-02-18T16:48:36Z

test/libsolidity/semanticTests/smoke_test.sol

+// g(uint256,uint256): 1, -2 -> 3
+// h(), 1 ether -> 1
+// j() -> FAILURE
+// i() # Does not exist. # -> FAILURE # Reverts. #


Is this correct? Why not -> FAILURE?

That's what it says I think - The second # marks the end of an inner-line-comment I think.

chriseth suggested changes Jan 28, 2019

View reviewed changes

erak self-assigned this Jan 28, 2019

erak force-pushed the semantic-tests-split branch from 077b788 to db34d02 Compare February 1, 2019 07:39

erak force-pushed the test-file-parser branch 6 times, most recently from 0fdfefc to 83eb987 Compare February 2, 2019 15:14

erak mentioned this pull request Feb 4, 2019

Test file parser. #5860

Merged

erak force-pushed the test-file-parser branch 2 times, most recently from ad80816 to f5fa62d Compare February 5, 2019 12:57

erak force-pushed the semantic-tests-split branch 6 times, most recently from ac13d28 to 4e34426 Compare February 6, 2019 13:13

ethereum deleted a comment from stackenbotten Feb 6, 2019

erak force-pushed the semantic-tests-split branch from 4e34426 to 3e173ac Compare February 6, 2019 15:51

erak force-pushed the test-file-parser branch from bda5619 to 9c97937 Compare February 6, 2019 15:56

erak force-pushed the semantic-tests-split branch from 3e173ac to 8db2dfa Compare February 6, 2019 15:57

erak force-pushed the test-file-parser branch from 9c97937 to a10982e Compare February 6, 2019 15:59

erak force-pushed the semantic-tests-split branch from 8db2dfa to bde513d Compare February 6, 2019 16:03

erak force-pushed the test-file-parser branch from a10982e to c9c4578 Compare February 6, 2019 16:26

erak force-pushed the semantic-tests-split branch 2 times, most recently from 5a4e9e5 to b5a8662 Compare February 6, 2019 16:55

ekpyron requested changes Feb 7, 2019

View reviewed changes

erak changed the base branch from test-file-parser to develop February 11, 2019 10:08

erak force-pushed the semantic-tests-split branch from b5a8662 to 6ac44b9 Compare February 11, 2019 22:31

Implements infrastructure for semantic tests.

190634e

erak force-pushed the semantic-tests-split branch from 6ac44b9 to 90289ee Compare February 14, 2019 16:28

erak commented Feb 14, 2019

View reviewed changes

test/RPCSession.cpp Show resolved Hide resolved

erak commented Feb 14, 2019

View reviewed changes

test/libsolidity/SemanticTest.cpp Show resolved Hide resolved

ekpyron previously approved these changes Feb 15, 2019

View reviewed changes

test/libsolidity/SemanticTest.cpp Outdated Show resolved Hide resolved

Adds semantic tests to test framework and isoltest.

dacad62

erak dismissed ekpyron’s stale review via dacad62 February 15, 2019 12:29

erak force-pushed the semantic-tests-split branch from 56b9874 to dacad62 Compare February 15, 2019 12:29

ekpyron approved these changes Feb 15, 2019

View reviewed changes

chriseth reviewed Feb 18, 2019

View reviewed changes

chriseth merged commit 968ca88 into develop Feb 18, 2019

ekpyron deleted the semantic-tests-split branch February 18, 2019 17:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Semantic test infrastructure #5872

Semantic test infrastructure #5872

erak commented Jan 27, 2019 •

edited by ekpyron

Loading

codecov bot commented Jan 28, 2019 •

edited

Loading

ekpyron left a comment

ekpyron Feb 7, 2019

erak Feb 11, 2019

ekpyron Feb 12, 2019

erak Feb 14, 2019

erak Feb 14, 2019

Marenz Feb 18, 2019

erak commented Feb 15, 2019

erak commented Feb 15, 2019

ekpyron left a comment

chriseth Feb 18, 2019

ekpyron Feb 18, 2019

chriseth Feb 18, 2019

	Usage: isoltest [Options] --testpath path --ipcpath ipcpath
	Usage: isoltest [Options] --testpath path {--ipcpath ipcpath, --no-ipc}

Semantic test infrastructure #5872

Semantic test infrastructure #5872

Conversation

erak commented Jan 27, 2019 • edited by ekpyron Loading

Checklist

codecov bot commented Jan 28, 2019 • edited Loading

Codecov Report

ekpyron left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erak commented Feb 15, 2019

erak commented Feb 15, 2019

ekpyron left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erak commented Jan 27, 2019 •

edited by ekpyron

Loading

codecov bot commented Jan 28, 2019 •

edited

Loading