Add support for token dictionaries #336
Conversation
| _, err := os.Stat(*flagDict) | ||
| if err != nil { | ||
| // If not it might be because a dictLevel was provided by appending @<num> to the dict path | ||
| atIndex := strings.LastIndex(*flagDict, "@") |
There was a problem hiding this comment.
It seems that this logic can be simpler if we try to split by @ first. Otherwise we have too many branches and duplicated error handling. If there is no specific reason to try to open the file with "@" first, please split by @ before first stat.
| } | ||
| dictLevel, err = strconv.Atoi((*flagDict)[atIndex+1:]) | ||
| if err != nil { | ||
| log.Printf("could not convert dict level using dict level 0 instead") |
There was a problem hiding this comment.
Why not Fatalf? That's incorrect user input.
| @@ -57,6 +63,32 @@ func main() { | |||
| log.Fatalf("both -http and -worker are specified") | |||
| } | |||
|
|
|||
| if *flagDict != "" { | |||
There was a problem hiding this comment.
Please move this logic to a helper function. It's too low level for main.
| @@ -116,6 +216,35 @@ func newHub(metadata MetaData) *Hub { | |||
| ro.intLits = append(ro.intLits, []byte(lit.Val)) | |||
| } | |||
| } | |||
|
|
|||
| if dictPath != "" { | |||
| /* | |||
There was a problem hiding this comment.
Please use C-style single-line // comments.
| } | ||
| token := parseDictTokenLine(&tokenLine, tokenLineNo) | ||
| if token != nil { | ||
| // add token to ro.strLits |
There was a problem hiding this comment.
This does not look useful. Remove.
| @@ -73,6 +77,102 @@ type Stats struct { | |||
| restarts uint64 | |||
| } | |||
|
|
|||
| func parseDictTokenLine(tokenLine *[]byte, tokenLineNo int) *[]byte { | |||
There was a problem hiding this comment.
Please move this function to somewhere at the bottom (after use). It's not the most important function of the file to be first.
| if bytes.HasPrefix(bytes.TrimSpace(tokenLine), []byte("#")) || len(tokenLine) == 0 { | ||
| continue | ||
| } | ||
| token := parseDictTokenLine(&tokenLine, tokenLineNo) |
| @@ -73,6 +77,102 @@ type Stats struct { | |||
| restarts uint64 | |||
| } | |||
|
|
|||
| func parseDictTokenLine(tokenLine *[]byte, tokenLineNo int) *[]byte { | |||
| token = append(token, (*tokenLine)[index]) | ||
| break | ||
|
|
||
| case byte('x'): |
There was a problem hiding this comment.
Do these need to be casted to byte?
Hey,
We have had an internship project in Trail of Bits to improve go-fuzz recently which was done by @vfsrfs.
We are aware of the ongoing official work on native fuzzing support but since we still rely on go-fuzz we went ahead to improve its pain points and so that's why we propose this PR. Feel free to drop it if you feel it is too much or you do not want to introduce any changes in go-fuzz.
Below I am pasting the description from the original PR merged to our fork of go-fuzz (trail-of-forks#2).
This PR adds support for dictionaries containing interesting keywords (tokens) that are useful for the mutation of inputs while fuzzing, particularly, when fuzzing syntax-aware programs (#174). This modification allows to provide the
-dictflag to go-fuzz, so that the user can provide a dictionary file with useful tokens for the fuzzing campaign. E.g.:The tokens parsed from the dictionary are stored in
ROData.strLits, as those are the string literals that are used by the mutator engine when generating new fuzzing inputs.The dictionary format that is accepted by the
-dictflag is the same that is used by AFL/Libfuzzer (see https://github.com/google/AFL/tree/master/dictionaries).This dictionary format defines that there is one token per line. Every line consists of a name followed by an equal sign and the token in quotes (e.g.
name=”token”). It is also possible to define binary sequences by providing the values in hex (e.g.\xNN) within the token. To insert a backslash or a double quote within the token, it has to be escaped using a backslash (e.g.\\or\”).\nand\tare recognized as well, since they might be useful for text-based protocols. Other problematic characters can be added by providing its hex value.To make this implementation fully compatible with AFL/Libfuzzer’s dictionaries, token levels are supported. A level can be appended to every token, by appending
@<num>to the keyword, e.g.keyword@1=”token”These tokens will be loaded only, if the dictionary level is equal to or greater than the specified number. The default dictionary level is 0, but it can be increased by appending
@<num>to the dictionary path. E.g.: