Source coverage printing after fuzzing campaign #516

ggrieco-tob · 2020-09-28T22:37:58Z

This PR allows Echidna to use source information from hevm to print which lines were covered during a fuzzing campaign. There is no need to configure anything, the source information is included in the crytic-compile output. The user should only enable coverage and corpusDir. However, the user must update to crytic-compile 0.1.10 (or newer) to make sure the source files are properly sorted when compiling.

The directly structure for the corpus directly was slight changed. For instance, after running flags.sol, the corpus directory will looks like this:

$ ls corpus/ -R
corpus/:
coverage  covered.txt

corpus/coverage:
-4293026413532156583.txt  4589328475588012129.txt

where the covered.txt file looks like this:

examples/solidity/basic/flags.sol
*r  |contract Test {
  bool private flag0 = true;
  bool private flag1 = true;

*   |  function set0(int val) public returns (bool){
*   |    if (val % 100 == 0)
*   |      flag0 = false;
  }

*   |  function set1(int val) public returns (bool){
*   |    if (val % 10 == 0 && !flag0)
*   |      flag1 = false;
  }

  function echidna_alwaystrue() public returns (bool){
    return(true);
  }

  function echidna_revert_always() public returns (bool){
    revert();
  }

  function echidna_sometimesfalse() public returns (bool){
    return(flag1);
  }

}

Constructors (#539) and echidna properties are not counting coverage. This should be fixed in another PR.

Each reachable line will have zero or more 'markers', signaling if how the sequence of transactions ended when covering that line. So far, we have support for the following markers:

* for sequences ended with a return.
r for sequences ended with a revert.
o for sequences ended with an out-of-gas error.
e for every other type of error (e.g. self-destruction, insufficient balance, etc).

It also introduced a small refactoring to move the corpus functions into a separated module inside Output.

lib/Echidna/Exec.hs

lib/Echidna/Solidity.hs

lib/Echidna/Output/Source.hs

arcz · 2020-11-24T22:28:24Z

lib/Echidna.hs

@@ -39,14 +40,14 @@ import qualified Data.List.NonEmpty as NE
 -- * A list of transaction sequences to initialize the corpus
 prepareContract :: (MonadCatch m, MonadRandom m, MonadReader x m, MonadIO m, MonadFail m,
                    Has TxConf x, Has SolConf x)
-                => EConfig -> NE.NonEmpty FilePath -> Maybe String -> Seed -> m (VM, World, [SolTest], Maybe GenDict, [[Tx]])
+                => EConfig -> NE.NonEmpty FilePath -> Maybe String -> Seed -> m (VM, SourceCache, [SolcContract], World, [SolTest], Maybe GenDict, [[Tx]])


maybe prepareContract should take [SolcContract] provided by prior calling of Echidna.Solidity.contracts as an argument rather than FilePaths?

What do you mean exactly?

lib/Echidna/Output/Source.hs

lib/Echidna/Solidity.hs

…dev-sources

incertia · 2021-02-03T17:10:41Z

lib/Echidna/Campaign.hs

@@ -161,7 +161,7 @@ execTxOptC t = do
  res <- execTxWith vmExcept (usingCoverage $ pointCoverage cov) t
  let vmr = getResult $ fst res
  -- Update the coverage map with the proper binary according to the vm result
-  cov %= mapWithKey (\_ s -> DS.map (set _2 vmr) s)
+  cov %= mapWithKey (\_ s -> DS.map (\(i, j, _) -> (i, j, vmr)) s)


incertia · 2021-02-03T17:11:48Z

lib/Echidna/Exec.hs


 -- | Capture the current PC and bytecode (without metadata). This should identify instructions uniquely.
 pointCoverage :: (MonadState x m, Has VM x) => Lens' x CoverageMap -> m ()
 pointCoverage l = do
  v <- use hasLens
-  l %= M.insertWith (const . S.insert $ (v ^. state . pc, Success))
+  l %= M.insertWith (const . S.insert $ ( v ^. state . pc, fromJust $ vmOpIx v, Success))


rogue space

incertia · 2021-02-03T17:13:25Z

lib/Echidna/Output/JSON.hs

@@ -22,7 +22,7 @@ data Campaign = Campaign
  , _error :: Maybe String
  , _tests :: [Test]
  , seed :: Int
-  , coverage :: Map String [(Int, TxResult)]
+  , coverage :: Map String [(Int, Int, TxResult)]


use your new types?

incertia · 2021-02-03T17:13:40Z

lib/Echidna/Output/Source.hs

@@ -0,0 +1,122 @@
+{-# LANGUAGE DataKinds #-}


incertia · 2021-02-03T17:14:16Z

lib/Echidna/Output/Source.hs

+
+-- | Mark one particular line, from a list of lines, keeping the order of them 
+markLine :: Int -> TxResult -> FilePathText -> [(FilePathText, Text)] -> [(FilePathText, Text)] 
+markLine n r cf ls = case splitAt (n-1) ls of


let's make this multiline and indent a bit less

I think now it is better

incertia · 2021-02-03T17:20:08Z

lib/Echidna/Output/Corpus.hs

+
+import Echidna.Types.Tx
+
+listDirectory :: FilePath -> IO [FilePath]


maybe move these directory functions to a util module

incertia · 2021-02-03T17:21:09Z

lib/Echidna/Output/Source.hs

+
+type FilePathText = Text
+
+saveCoveredCode :: Maybe FilePath -> SourceCache -> [SolcContract] -> CoverageMap -> IO ()


maybe rename this to saveCoverage

incertia · 2021-02-03T17:22:51Z

src/test/Common.hs

@@ -73,7 +73,7 @@ runContract :: FilePath -> Maybe ContractName -> EConfig -> IO Campaign
 runContract f c cfg =
  flip runReaderT cfg $ do
    g <- getRandom
-    (v, w, ts, d, txs) <- prepareContract cfg (f :| []) c g
+    (v, _, _, w, ts, d, txs) <- prepareContract cfg (f :| []) c g
    -- start ui and run tests
    campaign (pure ()) v w ts d txs


now that we have a bunch of things from prepareContract can we make a custom type for prepareContract and campaign and ui to ingest the custom type instead of adding a new parameter every time? feel free to make an issue and refactor in a separate PR.

Create a new issue to track down this.

incertia · 2021-02-03T17:23:21Z

lib/Echidna/Types/Signature.hs

 import Data.List.NonEmpty (NonEmpty)
+import Data.ByteString (ByteString)


alphabetize these imports

incertia · 2021-02-03T17:23:43Z

lib/Echidna/Transaction.hs

 import Control.Monad.Random.Strict (MonadRandom, getRandomR, uniform)
 import Control.Monad.Reader.Class (MonadReader)
 import Control.Monad.State.Strict (MonadState, State, runState, get, put)
-import Data.Aeson (ToJSON(..), decodeStrict, encodeFile)
+import Control.Monad (join, liftM2)


alphabetize imports

incertia and others added 8 commits September 9, 2020 11:34

hevm-0.41.x

0bd6f75

Buffer.hs

267f099

fix some non-exhaustive patterns

6e28492

actually implement viewBuffer lol

4a7ae97

disable gas price usage until hevm fixes its issue

fa5ff67

proof-of-concept of coverage source printing

f72f0db

fixes

770b6eb

fixed bugs

817b226

ggrieco-tob mentioned this pull request Oct 26, 2020

Feature request: some visibility into passing tests call sequences #534

Closed

ggrieco-tob added 14 commits November 2, 2020 10:36

merge + restored tests

f902e5e

upgrade to hevm-0.42 and fix tests

2f79a75

fixed test + default value

cc8937a

fixed test + default value

0cbb34f

merge

82f737f

improvements

60d7dde

refactoring

55d3a24

merge

3026f59

hlint fixes

68b5289

hlint fixes

6e2f116

more fixes

eed11e7

missing file

3f4693f

merge

33acfb6

fixed tests

442a14f

ggrieco-tob mentioned this pull request Nov 16, 2020

some simplifications #513

Merged

ggrieco-tob and others added 5 commits November 17, 2020 14:01

Avoid using showHex with negative values

1911401

Update Processor.hs

fae6fd8

fix

2420d4b

merge

83ccdaf

Merge remote-tracking branch 'origin/fix-showHex' into dev-sources

93ffd96

ggrieco-tob mentioned this pull request Nov 18, 2020

debugger support #483

Open

ggrieco-tob added 3 commits November 23, 2020 14:44

clean-up

320ccf2

merge

6de1593

fixes

3048335

ggrieco-tob marked this pull request as ready for review November 23, 2020 18:28

small changes to flags.sol

c7b3c42

arcz reviewed Nov 24, 2020

View reviewed changes

ggrieco-tob added 9 commits December 1, 2020 10:24

improved signature maps

6e638eb

Merge remote-tracking branch 'origin/dev-better-signature-maps' into …

4cdf0f1

…dev-sources

fix for source info collection

5dd2293

fix

c283a0f

fixes

faeeeb9

merge

38c9354

fixes

22db4f5

merge

160e48e

merge

b2c9ad0

incertia suggested changes Feb 3, 2021

View reviewed changes

ggrieco-tob mentioned this pull request Feb 8, 2021

Type refactoring for prepareContract campaign and ui #610

Closed

fixes

f8bf360

ggrieco-tob merged commit 4e0a217 into master Feb 8, 2021

ggrieco-tob deleted the dev-sources branch February 8, 2021 21:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Source coverage printing after fuzzing campaign #516

Source coverage printing after fuzzing campaign #516

ggrieco-tob commented Sep 28, 2020 •

edited

Loading

arcz Nov 24, 2020

ggrieco-tob Feb 2, 2021

incertia Feb 3, 2021

ggrieco-tob Feb 8, 2021

incertia Feb 3, 2021

ggrieco-tob Feb 8, 2021

incertia Feb 3, 2021

incertia Feb 3, 2021

ggrieco-tob Feb 8, 2021

incertia Feb 3, 2021

ggrieco-tob Feb 8, 2021

incertia Feb 3, 2021

ggrieco-tob Feb 8, 2021

incertia Feb 3, 2021

ggrieco-tob Feb 8, 2021

incertia Feb 3, 2021

ggrieco-tob Feb 8, 2021

incertia Feb 3, 2021

ggrieco-tob Feb 8, 2021

incertia Feb 3, 2021

ggrieco-tob Feb 8, 2021


		import Echidna.Types.Tx

		listDirectory :: FilePath -> IO [FilePath]


		type FilePathText = Text

		saveCoveredCode :: Maybe FilePath -> SourceCache -> [SolcContract] -> CoverageMap -> IO ()

		import Data.List.NonEmpty (NonEmpty)
		import Data.ByteString (ByteString)

Source coverage printing after fuzzing campaign #516

Source coverage printing after fuzzing campaign #516

Conversation

ggrieco-tob commented Sep 28, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ggrieco-tob commented Sep 28, 2020 •

edited

Loading