Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

regression task suggestions #1

Closed
wants to merge 15 commits into from

Conversation

bmc-msft
Copy link

As is, the regression task implements something primarily useful to answer "does this build have this bug". However, this makes service-level integration of "tell me what bugs I've fixed" difficult.

This PR changes the flow slightly always creating a RegressionReport regardless of outcome. On repeating the crash, this report will include crash_test_result.crash_result. On not repeating the crash, this report will include crash_test_result.no_repro.

This probably impacts the git bisect workflow, but enables the tell me if we fixed a bug workflow in addition to tell me if we still have a bug workflow.

To demonstrate this idea, I added libfuzzer-regression test binaries of broken.exe and fixed.exe (not yet included in integration testing). As you would expect by the name, broken.exe has a bug and fixed.exe does not.

We can demonstrate this by doing the following:

PROJ=regress
NAME=regress
onefuzz template libfuzzer basic --vm_count 1 --duration 1 --target_exe ./broken.exe $PROJ $NAME 1 linux --wait_for_files unique_reports
onefuzz template stop ${PROJ} ${NAME}
onefuzz template libfuzzer basic --vm_count 1 --duration 1 --target_exe ./fixed.exe $PROJ $NAME 2 linux --wait_for_files regression_reports 
onefuzz template stop ${PROJ} ${NAME}
onefuzz template libfuzzer basic --vm_count 1 --duration 1 --target_exe ./broken.exe $PROJ $NAME 3 linux --wait_for_files regression_reports

During the first job, we'll see a single unique crash report created and no regression reports.

During the second job, we'll still have that unique one crash report. In the job specific regression reports container, now we'll have a regression report. Upon pulling that regression report, we'll see that the crash_test_result.no_repro exists.

During the third job, we'll still have that unique one crash report. In the job specific regression reports container, now we'll have a regression report. Upon pulling that regression report, we'll see that the crash_test_result.crash_result exists.

@bmc-msft bmc-msft closed this Mar 11, 2021
chkeita pushed a commit that referenced this pull request Mar 19, 2021
For a given entry in a call stack, this parses out the following: line, function name, function offset, source file name, source file line, module path, and module offset.

Additionally, this provides a code-generated libclusterfuzz port of the regular expressions used for stack minimization.

For an example of the minimization, instead of:
```json
[
"#0 0x56512a9c1418 in __sanitizer_print_stack_trace /b/s/w/ir/cache/builder/src/third_party/llvm/compiler-rt/lib/asan/asan_stack.cpp:86:3",
"#1 0x56512aaaa42d in fuzzer::PrintStackTrace() third_party/libFuzzer/src/FuzzerUtil.cpp:205:5",
"#2 0x56512aa6a85e in fuzzer::Fuzzer::CrashCallback() third_party/libFuzzer/src/FuzzerLoop.cpp:232:3",
"#3 0x56512aa6a7df in fuzzer::Fuzzer::StaticCrashSignalCallback() third_party/libFuzzer/src/FuzzerLoop.cpp:203:6",
"#4 0x56512aaab948 in fuzzer::CrashHandler(int, siginfo_t*, void*) third_party/libFuzzer/src/FuzzerUtilPosix.cpp:46:3",
"#5 0x7f1ee3f0188f  (/lib/x86_64-linux-gnu/libpthread.so.0+0x1288f)",
"#6 0x56512a9e5aa1 in Json::OurReader::parse(char const*, char const*, Json::Value&, bool) third_party/jsoncpp/source/src/lib_json/json_reader.cpp:1062:10",
"#7 0x56512a9eedb4 in Json::OurCharReader::parse(char const*, char const*, Json::Value*, std::__Cr::basic_string<char, std::__Cr::char_traits<char>, std::__Cr::allocator<char> >*) third_party/jsoncpp/source/src/lib_json/json_reader.cpp:1899:23",
"#8 0x56512a9e03a3 in LLVMFuzzerTestOneInput third_party/jsoncpp/fuzzers/json_fuzzer.cc:39:24",
"#9 0x56512aa6d0cf in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) third_party/libFuzzer/src/FuzzerLoop.cpp:556:15",
"#10 0x56512aa3b7da in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) third_party/libFuzzer/src/FuzzerDriver.cpp:292:6",
"#11 0x56512aa4108a in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) third_party/libFuzzer/src/FuzzerDriver.cpp:774:9","#12 0x56512aa821ac in main third_party/libFuzzer/src/FuzzerMain.cpp:19:10",
"#13 0x7f1ee3361b96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310",
]
```

The minimized call stack is:
```json
[
"Json::OurReader::parse(char const*, char const*, Json::Value&, bool)", 
"Json::OurCharReader::parse(char const*, char const*, Json::Value*, std::__Cr::basic_string<char, std::__Cr::char_traits<char>, std::__Cr::allocator<char> >*)",
"json_fuzzer.cc"
]
```

This also provides a naïve function name list, which comes close to Clusterfuzz's function identification.

This would result in:
```json
[
    "Json::OurReader::parse",
    "Json::OurCharReader::parse",
    "json_fuzzer.cc"
]
```

Lastly, for our `stack hash` functionality used by the crash reporting task, those now provide the ability to specify the number of frames to include when building the hash.
bmc-msft added a commit that referenced this pull request Mar 22, 2021
Adds `test-input` and `test-input-libfuzzer`, which print the CrashTestResult in json form.

While many of the existing tasks make sense running in a managed loop, crash report generation is something that having a single one-off is useful.

Example:
```
$ onefuzz-agent local test-input /tmp/fuzz.exe /tmp/crash.txt
{
  "crash_report": {
    "input_sha256": "a35b3ce1038750e9175a6dcd3f64c8d4e85720affb12cc11f5d0b6889274d06e",
    "executable": "/tmp/fuzz.exe",
    "crash_type": "SIGABRT",
    "crash_site": "0x7f0d9d4ad18b in gsignal+0xcb (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x4618b)",
    "call_stack": [
      "#0 0x7f0d9d4ad18b in gsignal+0xcb (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x4618b)",
      "#1 0x7f0d9d48c859 in abort+0x12b (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x25859)",
      "#2 0x7f0d9d4f73ee in <unknown> (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x903ee)",
      "#3 0x7f0d9d599b4a in __fortify_fail+0x2a (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x132b4a)",
      "#4 0x7f0d9d5983e6 in __chk_fail+0x16 (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x1313e6)",
      "#5 0x7f0d9d597e09 in __strncpy_chk+0x19 (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x130e09)",
      "#6 0x400a54 in from_file+0xa4 (/tmp/fuzz.exe+0xa54)",
      "#7 0x7f0d9d48e0b3 in __libc_start_main+0xf3 (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x270b3)",
      "#8 0x40077a in _start+0x2a (/tmp/fuzz.exe+0x77a)"
    ],
    "call_stack_sha256": "6906234fb235690cc2843a1a55f49ff68b424e54bec55f9b8258415d97b3e638",
    "task_id": "00000000-0000-0000-0000-000000000000",
    "job_id": "00000000-0000-0000-0000-000000000000"
  }
}
$
```
bmc-msft added a commit that referenced this pull request Mar 23, 2021
This builds upon microsoft#591 to expand the stack minimization to crash reporting mechanisms.

Example (see microsoft#703 for an example without the new functionality):

```
$ onefuzz-agent local test-input /tmp/fuzz.exe /etc/passwd
{
  "crash_report": {
    "input_sha256": "a35b3ce1038750e9175a6dcd3f64c8d4e85720affb12cc11f5d0b6889274d06e",
    "executable": "/tmp/fuzz.exe",
    "crash_type": "SIGABRT",
    "crash_site": "0x7ffff7e0d18b in gsignal+0xcb (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x4618b)",
    "call_stack": [
      "#0 0x7ffff7e0d18b in gsignal+0xcb (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x4618b)",
      "#1 0x7ffff7dec859 in abort+0x12b (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x25859)",
      "#2 0x7ffff7e573ee in <unknown> (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x903ee)",
      "#3 0x7ffff7ef9b4a in __fortify_fail+0x2a (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x132b4a)",
      "#4 0x7ffff7ef83e6 in __chk_fail+0x16 (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x1313e6)",
      "#5 0x7ffff7ef7e09 in __strncpy_chk+0x19 (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x130e09)",
      "#6 0x400a54 in from_file+0xa4 (/tmp/fuzz.exe+0xa54)",
      "#7 0x7ffff7dee0b3 in __libc_start_main+0xf3 (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x270b3)",
      "#8 0x40077a in _start+0x2a (/tmp/fuzz.exe+0x77a)"
    ],
    "call_stack_sha256": "99625a7c103136e02910b65c7b60f1bbd1a7612242d6838da52d968369039409",
    "minimized_stack": [
      "__fortify_fail",
      "__chk_fail",
      "from_file"
    ],
    "minimized_stack_sha256": "237f13bfa384c6c2bc06369099373efbb36995a9ad00fd5469d354b5fc672ba1",
    "minimized_stack_function_names": [
      "__fortify_fail",
      "__chk_fail",
      "from_file"
    ],
    "minimized_stack_function_names_sha256": "237f13bfa384c6c2bc06369099373efbb36995a9ad00fd5469d354b5fc672ba1",
    "asan_log": "",
    "task_id": "00000000-0000-0000-0000-000000000000",
    "job_id": "00000000-0000-0000-0000-000000000000"
  }
}
$
```
bmc-msft pushed a commit that referenced this pull request Jun 24, 2021
The debug report created by the command `onefuzz debug  notification job <job id>`  is causing a crash in the regression task 
```
error running task: libfuzzer regression

Caused by:
    0: handling crash reports
    1: unable to parse crash report: fake-crash-sample.json
    2: unable to parse report: task_unique_reports_2/fake-crash-sample.json - "{\"input_url\": null, \"input_blob\": {\"account\": \"fuzz27ee6imdmr5gy\", \"container\": \"oft-crashes-cecbd958a1f257688f9768edaaf6c94d\", \"name\": \"fake-crash-sample\"}, \"executable\": \"fuzz.exe\", \"crash_type\": \"fake crash report\", \"crash_site\": \"fake crash site\", \"call_stack\": [\"#0 fake\", \"#1 call\", \"#2 stack\"], \"call_stack_sha256\": \"0000000000000000000000000000000000000000000000000000000000000000\", \"input_sha256\": \"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\", \"asan_log\": \"fake asan log\", \"task_id\": \"b1107de0-c3cb-43ff-ab68-5accc579f4d4\", \"job_id\": \"afa45e3e-9a75-4a47-8d59-ef3154599fc7\", \"scariness_score\": null, \"scariness_description\": null, \"minimized_stack\": null, \"minimized_stack_sha256\": null, \"minimized_stack_function_names\": null, \"minimized_stack_function_names_sha256\": null}"
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants