Transition to contextualized `Done` state on setup error #1

ranweiler · 2020-09-22T22:05:53Z

Summary of the Pull Request

When target setup fails, due to IO or a script, transition to an enriched Done scheduler state, rather than terminating the agent.

NOTE: based on pending branch, retarget before final review.

PR Checklist

Applies to work item: Bump flume from 0.10.8 to 0.10.9 in /src/agent #13
CLA signed. If not, go over here and sign the CLI.
Tests added/passed
Requires documentation to be updated
I've discussed this with core contributors already. If not checked, I'm ready to accept this work might be rejected in favor of a different grand plan.

Validation Steps Performed

Create a job with a setup script that intentionally exits nonzero. The agent should emit a Done event. Task and job state should be updated. Node should be automatically reimaged.

Change the internal `ISetupRunner` interface to return a serializable representation of the script process output. Allow callers to decide how to handle a non-success exit disposition.

For a given entry in a call stack, this parses out the following: line, function name, function offset, source file name, source file line, module path, and module offset. Additionally, this provides a code-generated libclusterfuzz port of the regular expressions used for stack minimization. For an example of the minimization, instead of: ```json [ "#0 0x56512a9c1418 in __sanitizer_print_stack_trace /b/s/w/ir/cache/builder/src/third_party/llvm/compiler-rt/lib/asan/asan_stack.cpp:86:3", "#1 0x56512aaaa42d in fuzzer::PrintStackTrace() third_party/libFuzzer/src/FuzzerUtil.cpp:205:5", "#2 0x56512aa6a85e in fuzzer::Fuzzer::CrashCallback() third_party/libFuzzer/src/FuzzerLoop.cpp:232:3", "#3 0x56512aa6a7df in fuzzer::Fuzzer::StaticCrashSignalCallback() third_party/libFuzzer/src/FuzzerLoop.cpp:203:6", "#4 0x56512aaab948 in fuzzer::CrashHandler(int, siginfo_t*, void*) third_party/libFuzzer/src/FuzzerUtilPosix.cpp:46:3", "#5 0x7f1ee3f0188f (/lib/x86_64-linux-gnu/libpthread.so.0+0x1288f)", "#6 0x56512a9e5aa1 in Json::OurReader::parse(char const*, char const*, Json::Value&, bool) third_party/jsoncpp/source/src/lib_json/json_reader.cpp:1062:10", "#7 0x56512a9eedb4 in Json::OurCharReader::parse(char const*, char const*, Json::Value*, std::__Cr::basic_string<char, std::__Cr::char_traits<char>, std::__Cr::allocator<char> >*) third_party/jsoncpp/source/src/lib_json/json_reader.cpp:1899:23", "#8 0x56512a9e03a3 in LLVMFuzzerTestOneInput third_party/jsoncpp/fuzzers/json_fuzzer.cc:39:24", "#9 0x56512aa6d0cf in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) third_party/libFuzzer/src/FuzzerLoop.cpp:556:15", "#10 0x56512aa3b7da in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) third_party/libFuzzer/src/FuzzerDriver.cpp:292:6", "#11 0x56512aa4108a in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) third_party/libFuzzer/src/FuzzerDriver.cpp:774:9","#12 0x56512aa821ac in main third_party/libFuzzer/src/FuzzerMain.cpp:19:10", "#13 0x7f1ee3361b96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310", ] ``` The minimized call stack is: ```json [ "Json::OurReader::parse(char const*, char const*, Json::Value&, bool)", "Json::OurCharReader::parse(char const*, char const*, Json::Value*, std::__Cr::basic_string<char, std::__Cr::char_traits<char>, std::__Cr::allocator<char> >*)", "json_fuzzer.cc" ] ``` This also provides a naïve function name list, which comes close to Clusterfuzz's function identification. This would result in: ```json [ "Json::OurReader::parse", "Json::OurCharReader::parse", "json_fuzzer.cc" ] ``` Lastly, for our `stack hash` functionality used by the crash reporting task, those now provide the ability to specify the number of frames to include when building the hash.

Adds `test-input` and `test-input-libfuzzer`, which print the CrashTestResult in json form. While many of the existing tasks make sense running in a managed loop, crash report generation is something that having a single one-off is useful. Example: ``` $ onefuzz-agent local test-input /tmp/fuzz.exe /tmp/crash.txt { "crash_report": { "input_sha256": "a35b3ce1038750e9175a6dcd3f64c8d4e85720affb12cc11f5d0b6889274d06e", "executable": "/tmp/fuzz.exe", "crash_type": "SIGABRT", "crash_site": "0x7f0d9d4ad18b in gsignal+0xcb (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x4618b)", "call_stack": [ "#0 0x7f0d9d4ad18b in gsignal+0xcb (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x4618b)", "#1 0x7f0d9d48c859 in abort+0x12b (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x25859)", "#2 0x7f0d9d4f73ee in <unknown> (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x903ee)", "#3 0x7f0d9d599b4a in __fortify_fail+0x2a (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x132b4a)", "#4 0x7f0d9d5983e6 in __chk_fail+0x16 (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x1313e6)", "#5 0x7f0d9d597e09 in __strncpy_chk+0x19 (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x130e09)", "#6 0x400a54 in from_file+0xa4 (/tmp/fuzz.exe+0xa54)", "#7 0x7f0d9d48e0b3 in __libc_start_main+0xf3 (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x270b3)", "#8 0x40077a in _start+0x2a (/tmp/fuzz.exe+0x77a)" ], "call_stack_sha256": "6906234fb235690cc2843a1a55f49ff68b424e54bec55f9b8258415d97b3e638", "task_id": "00000000-0000-0000-0000-000000000000", "job_id": "00000000-0000-0000-0000-000000000000" } } $ ```

This builds upon microsoft#591 to expand the stack minimization to crash reporting mechanisms. Example (see microsoft#703 for an example without the new functionality): ``` $ onefuzz-agent local test-input /tmp/fuzz.exe /etc/passwd { "crash_report": { "input_sha256": "a35b3ce1038750e9175a6dcd3f64c8d4e85720affb12cc11f5d0b6889274d06e", "executable": "/tmp/fuzz.exe", "crash_type": "SIGABRT", "crash_site": "0x7ffff7e0d18b in gsignal+0xcb (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x4618b)", "call_stack": [ "#0 0x7ffff7e0d18b in gsignal+0xcb (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x4618b)", "#1 0x7ffff7dec859 in abort+0x12b (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x25859)", "#2 0x7ffff7e573ee in <unknown> (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x903ee)", "#3 0x7ffff7ef9b4a in __fortify_fail+0x2a (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x132b4a)", "#4 0x7ffff7ef83e6 in __chk_fail+0x16 (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x1313e6)", "#5 0x7ffff7ef7e09 in __strncpy_chk+0x19 (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x130e09)", "#6 0x400a54 in from_file+0xa4 (/tmp/fuzz.exe+0xa54)", "#7 0x7ffff7dee0b3 in __libc_start_main+0xf3 (/usr/lib/x86_64-linux-gnu/libc-2.31.so+0x270b3)", "#8 0x40077a in _start+0x2a (/tmp/fuzz.exe+0x77a)" ], "call_stack_sha256": "99625a7c103136e02910b65c7b60f1bbd1a7612242d6838da52d968369039409", "minimized_stack": [ "__fortify_fail", "__chk_fail", "from_file" ], "minimized_stack_sha256": "237f13bfa384c6c2bc06369099373efbb36995a9ad00fd5469d354b5fc672ba1", "minimized_stack_function_names": [ "__fortify_fail", "__chk_fail", "from_file" ], "minimized_stack_function_names_sha256": "237f13bfa384c6c2bc06369099373efbb36995a9ad00fd5469d354b5fc672ba1", "asan_log": "", "task_id": "00000000-0000-0000-0000-000000000000", "job_id": "00000000-0000-0000-0000-000000000000" } } $ ```

The debug report created by the command `onefuzz debug notification job <job id>` is causing a crash in the regression task ``` error running task: libfuzzer regression Caused by: 0: handling crash reports 1: unable to parse crash report: fake-crash-sample.json 2: unable to parse report: task_unique_reports_2/fake-crash-sample.json - "{\"input_url\": null, \"input_blob\": {\"account\": \"fuzz27ee6imdmr5gy\", \"container\": \"oft-crashes-cecbd958a1f257688f9768edaaf6c94d\", \"name\": \"fake-crash-sample\"}, \"executable\": \"fuzz.exe\", \"crash_type\": \"fake crash report\", \"crash_site\": \"fake crash site\", \"call_stack\": [\"#0 fake\", \"#1 call\", \"#2 stack\"], \"call_stack_sha256\": \"0000000000000000000000000000000000000000000000000000000000000000\", \"input_sha256\": \"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\", \"asan_log\": \"fake asan log\", \"task_id\": \"b1107de0-c3cb-43ff-ab68-5accc579f4d4\", \"job_id\": \"afa45e3e-9a75-4a47-8d59-ef3154599fc7\", \"scariness_score\": null, \"scariness_description\": null, \"minimized_stack\": null, \"minimized_stack_sha256\": null, \"minimized_stack_function_names\": null, \"minimized_stack_function_names_sha256\": null}" ```

ranweiler changed the title ~~Setup err~~ Transition to contextualized Done state on setup error Sep 22, 2020

ranweiler added 7 commits September 22, 2020 16:17

Emit setup script output from ISetupRunner (#23)

ab41b89

Change the internal `ISetupRunner` interface to return a serializable representation of the script process output. Allow callers to decide how to handle a non-success exit disposition.

Add cause to Done state

e6aa4fb

Add Done case to SetupDone

e0e1165

Simplify DoneCause

5ce72bb

Transition to Done on setup script error

35e3ea8

Remove unused important

6e3f7b0

Emit new Done event

63e85b2

ranweiler force-pushed the setup-err branch from 8a1d086 to 63e85b2 Compare September 22, 2020 23:23

ranweiler closed this Sep 22, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Transition to contextualized `Done` state on setup error #1

Transition to contextualized `Done` state on setup error #1

ranweiler commented Sep 22, 2020

Transition to contextualized Done state on setup error #1

Transition to contextualized Done state on setup error #1

Conversation

ranweiler commented Sep 22, 2020

Summary of the Pull Request

PR Checklist

Validation Steps Performed

Transition to contextualized `Done` state on setup error #1

Transition to contextualized `Done` state on setup error #1