Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fuzzer out of memory errors #192

Open
demurgos opened this issue Nov 10, 2019 · 3 comments
Open

Fuzzer out of memory errors #192

demurgos opened this issue Nov 10, 2019 · 3 comments

Comments

@demurgos
Copy link

demurgos commented Nov 10, 2019

Hi,
I am fuzzing the Rust implementation of swf-parser with cargo fuzz. The fuzzer is helpful and found some issues, but it crashes due to OOM (out of memory) errors about a third of the time. When looking into the input that caused this error, it seems that the OOM error is caused by the fuzzer itself, not the library.

Here is the fuzz target:

fuzz_target!(|data: &[u8]| {
  if let Some((swf_version, data)) = data.split_first() {
    let _ = swf_parser::complete::parse_tag(data, *swf_version);
  }
});

You can run it yourself by cloning the repo and then running:

cd rs
cargo fuzz run tag

The OOM is caused when the fuzzer generates the following inputs:

  • b"" (empty slice): Due to how the fuzz target is defined, the library shouldn't even be called. How is it possible to cause an OOM? This input is the most common cause of OOM.
  • b"\x5b\x01\x06\x00\x40" this input caused an OOM error once. This is a well-formed input that works fine when executed as a unit test (it produces a tag marking the SWF file as protected, with an empty password). There's no recursion or advanced resource management going on in the lib: this just produces a struct and should not exceed the default 2GiB memory limit.

Here is a log of one of the execution causing an OOM due to the empty slice:

See log
/home/demurgos/.cargo/bin/cargo fuzz run tag
       Fresh unicode-xid v0.1.0
       Fresh semver-parser v0.7.0
       Fresh cfg-if v0.1.7
       Fresh build_const v0.2.1
       Fresh version_check v0.1.5
       Fresh autocfg v0.1.6
       Fresh nodrop v0.1.14
       Fresh static_assertions v0.3.4
       Fresh ucd-util v0.1.3
       Fresh lazy_static v1.3.0
       Fresh cc v1.0.34
       Fresh utf8-ranges v1.0.2
       Fresh hex v0.3.2
       Fresh itoa v0.4.3
       Fresh adler32 v1.0.3
       Fresh half v1.4.0
       Fresh arbitrary v0.1.1
       Fresh semver v0.9.0
       Fresh log v0.4.6
       Fresh regex-syntax v0.6.6
       Fresh thread_local v0.3.6
       Fresh inflate v0.4.5
       Fresh proc-macro2 v0.4.27
       Fresh rustc_version v0.2.3
       Fresh memchr v2.2.0
       Fresh ryu v1.0.2
       Fresh arrayvec v0.4.12
       Fresh byteorder v1.3.1
       Fresh quote v0.6.11
       Fresh crc v1.8.1
       Fresh aho-corasick v0.7.6
       Fresh syn v0.15.30
       Fresh lzma-rs v0.1.1
       Fresh regex v1.1.9
       Fresh num-traits v0.2.8
       Fresh serde_derive v1.0.90
       Fresh lexical-core v0.4.6
       Fresh libfuzzer-sys v0.1.1
       Fresh serde v1.0.90
       Fresh nom v5.0.1
       Fresh swf-fixed v0.1.4
       Fresh serde_json v1.0.41
       Fresh swf-tree v0.9.0 (https://github.com/open-flash/swf-tree.git?rev=524f19adc098d83900e94bbe693887a63b50786d#524f19ad)
   Compiling swf-parser v0.9.0 (/data/projects/open-flash/swf-parser/rs)
     Running `rustc --edition=2018 --crate-name swf_parser /data/projects/open-flash/swf-parser/rs/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C debuginfo=2 -C metadata=61e283fed3cc1cf1 -C extra-filename=-61e283fed3cc1cf1 --out-dir /data/projects/open-flash/swf-parser/rs/fuzz/target/x86_64-unknown-linux-gnu/debug/deps --target x86_64-unknown-linux-gnu -C incremental=/data/projects/open-flash/swf-parser/rs/fuzz/target/x86_64-unknown-linux-gnu/debug/incremental -L dependency=/data/projects/open-flash/swf-parser/rs/fuzz/target/x86_64-unknown-linux-gnu/debug/deps -L dependency=/data/projects/open-flash/swf-parser/rs/fuzz/target/debug/deps --extern half=/data/projects/open-flash/swf-parser/rs/fuzz/target/x86_64-unknown-linux-gnu/debug/deps/libhalf-edb4f4d84be331bf.rmeta --extern inflate=/data/projects/open-flash/swf-parser/rs/fuzz/target/x86_64-unknown-linux-gnu/debug/deps/libinflate-5b679699785d38a5.rmeta --extern lazy_static=/data/projects/open-flash/swf-parser/rs/fuzz/target/x86_64-unknown-linux-gnu/debug/deps/liblazy_static-baf135a10834eeaf.rmeta --extern lzma_rs=/data/projects/open-flash/swf-parser/rs/fuzz/target/x86_64-unknown-linux-gnu/debug/deps/liblzma_rs-246e16db36b00216.rmeta --extern memchr=/data/projects/open-flash/swf-parser/rs/fuzz/target/x86_64-unknown-linux-gnu/debug/deps/libmemchr-9950ca4825b54084.rmeta --extern nom=/data/projects/open-flash/swf-parser/rs/fuzz/target/x86_64-unknown-linux-gnu/debug/deps/libnom-753393ed81863ed8.rmeta --extern num_traits=/data/projects/open-flash/swf-parser/rs/fuzz/target/x86_64-unknown-linux-gnu/debug/deps/libnum_traits-beec309a9be3db3c.rmeta --extern regex=/data/projects/open-flash/swf-parser/rs/fuzz/target/x86_64-unknown-linux-gnu/debug/deps/libregex-8590be608ed69cc2.rmeta --extern serde_json=/data/projects/open-flash/swf-parser/rs/fuzz/target/x86_64-unknown-linux-gnu/debug/deps/libserde_json-b650a2f567d8eed8.rmeta --extern swf_fixed=/data/projects/open-flash/swf-parser/rs/fuzz/target/x86_64-unknown-linux-gnu/debug/deps/libswf_fixed-a7f81cf33dce1629.rmeta --extern swf_tree=/data/projects/open-flash/swf-parser/rs/fuzz/target/x86_64-unknown-linux-gnu/debug/deps/libswf_tree-57a868c88b0df0ab.rmeta --cfg fuzzing -Cpasses=sancov -Cllvm-args=-sanitizer-coverage-level=4 -Cllvm-args=-sanitizer-coverage-trace-compares -Cllvm-args=-sanitizer-coverage-inline-8bit-counters -Cllvm-args=-sanitizer-coverage-trace-geps -Cllvm-args=-sanitizer-coverage-prune-blocks=0 -Cllvm-args=-sanitizer-coverage-pc-table -Zsanitizer=address -Cllvm-args=-sanitizer-coverage-stack-depth`
   Compiling swf-parser-fuzz v0.0.1 (/data/projects/open-flash/swf-parser/rs/fuzz)
     Running `rustc --edition=2018 --crate-name tag fuzz_targets/tag.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C debuginfo=2 -C metadata=8ef32776a5e8aad8 -C extra-filename=-8ef32776a5e8aad8 --out-dir /data/projects/open-flash/swf-parser/rs/fuzz/target/x86_64-unknown-linux-gnu/debug/deps --target x86_64-unknown-linux-gnu -C incremental=/data/projects/open-flash/swf-parser/rs/fuzz/target/x86_64-unknown-linux-gnu/debug/incremental -L dependency=/data/projects/open-flash/swf-parser/rs/fuzz/target/x86_64-unknown-linux-gnu/debug/deps -L dependency=/data/projects/open-flash/swf-parser/rs/fuzz/target/debug/deps --extern libfuzzer_sys=/data/projects/open-flash/swf-parser/rs/fuzz/target/x86_64-unknown-linux-gnu/debug/deps/liblibfuzzer_sys-b6f1eb7920522c41.rlib --extern swf_parser=/data/projects/open-flash/swf-parser/rs/fuzz/target/x86_64-unknown-linux-gnu/debug/deps/libswf_parser-61e283fed3cc1cf1.rlib --cfg fuzzing -Cpasses=sancov -Cllvm-args=-sanitizer-coverage-level=4 -Cllvm-args=-sanitizer-coverage-trace-compares -Cllvm-args=-sanitizer-coverage-inline-8bit-counters -Cllvm-args=-sanitizer-coverage-trace-geps -Cllvm-args=-sanitizer-coverage-prune-blocks=0 -Cllvm-args=-sanitizer-coverage-pc-table -Zsanitizer=address -Cllvm-args=-sanitizer-coverage-stack-depth -L native=/data/projects/open-flash/swf-parser/rs/fuzz/target/x86_64-unknown-linux-gnu/debug/build/libfuzzer-sys-aab13c62c2cc6b9d/out`
    Finished dev [unoptimized + debuginfo] target(s) in 12.43s
       Fresh unicode-xid v0.1.0
       Fresh semver-parser v0.7.0
       Fresh cfg-if v0.1.7
       Fresh build_const v0.2.1
       Fresh autocfg v0.1.6
       Fresh nodrop v0.1.14
       Fresh version_check v0.1.5
       Fresh ucd-util v0.1.3
       Fresh cc v1.0.34
       Fresh static_assertions v0.3.4
       Fresh lazy_static v1.3.0
       Fresh adler32 v1.0.3
       Fresh itoa v0.4.3
       Fresh utf8-ranges v1.0.2
       Fresh hex v0.3.2
       Fresh half v1.4.0
       Fresh arbitrary v0.1.1
       Fresh semver v0.9.0
       Fresh log v0.4.6
       Fresh regex-syntax v0.6.6
       Fresh thread_local v0.3.6
       Fresh inflate v0.4.5
       Fresh proc-macro2 v0.4.27
       Fresh rustc_version v0.2.3
       Fresh ryu v1.0.2
       Fresh memchr v2.2.0
       Fresh arrayvec v0.4.12
       Fresh byteorder v1.3.1
       Fresh quote v0.6.11
       Fresh aho-corasick v0.7.6
       Fresh crc v1.8.1
       Fresh syn v0.15.30
       Fresh num-traits v0.2.8
       Fresh regex v1.1.9
       Fresh lzma-rs v0.1.1
       Fresh libfuzzer-sys v0.1.1
       Fresh serde_derive v1.0.90
       Fresh lexical-core v0.4.6
       Fresh serde v1.0.90
       Fresh nom v5.0.1
       Fresh swf-fixed v0.1.4
       Fresh serde_json v1.0.41
       Fresh swf-tree v0.9.0 (https://github.com/open-flash/swf-tree.git?rev=524f19adc098d83900e94bbe693887a63b50786d#524f19ad)
       Fresh swf-parser v0.9.0 (/data/projects/open-flash/swf-parser/rs)
       Fresh swf-parser-fuzz v0.0.1 (/data/projects/open-flash/swf-parser/rs/fuzz)
    Finished dev [unoptimized + debuginfo] target(s) in 0.03s
     Running `fuzz/target/x86_64-unknown-linux-gnu/debug/tag -artifact_prefix=/data/projects/open-flash/swf-parser/rs/fuzz/artifacts/tag/ /data/projects/open-flash/swf-parser/rs/fuzz/corpus/tag`
INFO: Seed: 1676543060
INFO: Loaded 1 modules   (39836 inline 8-bit counters): 39836 [0x55aecfc8a170, 0x55aecfc93d0c),
INFO: Loaded 1 PC tables (39836 PCs): 39836 [0x55aecfc93d10,0x55aecfd2f6d0),
INFO:      216 files found in /data/projects/open-flash/swf-parser/rs/fuzz/corpus/tag
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: seed corpus: files: 216 min: 1b max: 6b total: 1015b rss: 3397Mb
#217	INITED cov: 5457 ft: 6648 corp: 189/879b lim: 4 exec/s: 0 rss: 3397Mb
#863	REDUCE cov: 5457 ft: 6648 corp: 189/878b lim: 4 exec/s: 0 rss: 3397Mb L: 5/6 MS: 1 EraseBytes-
	NEW_FUNC[1/2]: ==205169== ERROR: libFuzzer: out-of-memory (used: 3397Mb; limit: 2048Mb)
   To change the out-of-memory limit use -rss_limit_mb=<N>

MS: 4 ChangeByte-ChangeBinInt-ChangeBinInt-ChangeBit-; base unit: 5a796ce42572e152c35f0061fff1cd41eaede520


artifact_prefix='/data/projects/open-flash/swf-parser/rs/fuzz/artifacts/tag/'; Test unit written to /data/projects/open-flash/swf-parser/rs/fuzz/artifacts/tag/oom-da39a3ee5e6b4b0d3255bfef95601890afd80709
Base64:
SUMMARY: libFuzzer: out-of-memory

Process finished with exit code 77

System information:

  • OS: Linux 64bit (x86_64-unknown-linux-gnu)
  • Toolchain: nightly-2019-11-06-x86_64-unknown-linux-gnu
  • Cargo fuzz version: 0.5.4 (latest)

The issue may still lie in my lib, but I find it very unlikely given the inputs causing the OOM errors.

@demurgos
Copy link
Author

I found a workaround but there still seems to be a bug in the way the default value is computed.

When looking into the log of runs producing an error, we can see this info message:

INFO: seed corpus: files: 216 min: 1b max: 6b total: 1015b rss: 3397Mb

and then the error is:

	NEW_FUNC[1/2]: ==205169== ERROR: libFuzzer: out-of-memory (used: 3397Mb; limit: 2048Mb)
   To change the out-of-memory limit use -rss_limit_mb=<N>

It means that when running the fuzz command without any arguments, it picks a default rss value that is already higher than the limit. cargo fuzz should make sure that the default value is below the limit (or raise the limit?).

My workaround is to manually raise the limit:

fuzz run tag -- -rss_limit_mb=4096

@ghost
Copy link

ghost commented Feb 1, 2021

Is there a solution to this? I am also having the same problem where memory usage rises constantly after every run, but no memory leaks are reported so I assume this is because libfuzzer is using more memory after each run.

@cameronelliott
Copy link

It would be cool to get an explanation with some insight, if anyone on the project has a good handle on how
libfuzzer impacts RSS aside from the fuzzer target.

I wanted to understand, so I spun up a NOP fuzz_target to see what would happen.

fuzz_target!(|do_not_use: u8| {});

Here is some of the output during a fuzzing run.

#14	INITED cov: 21 ft: 21 corp: 1/1b exec/s: 0 rss: 36Mb
#2097152	pulse  cov: 21 ft: 21 corp: 1/1b lim: 4096 exec/s: 699050 rss: 194Mb
#4194304	pulse  cov: 21 ft: 21 corp: 1/1b lim: 4096 exec/s: 599186 rss: 351Mb
#8388608	pulse  cov: 21 ft: 21 corp: 1/1b lim: 4096 exec/s: 559240 rss: 618Mb
#16777216	pulse  cov: 21 ft: 21 corp: 1/1b lim: 4096 exec/s: 541200 rss: 620Mb
#33554432	pulse  cov: 21 ft: 21 corp: 1/1b lim: 4096 exec/s: 541200 rss: 622Mb
#67108864	pulse  cov: 21 ft: 21 corp: 1/1b lim: 4096 exec/s: 536870 rss: 622Mb

So, it seems the libfuzzer library does consume a good chunk of memory, until maybe it converges
to some steady state in the case of an empty fuzz_target.
I presume with larger more complex input state spaces the libfuzzer RSS use could dramatically increase.

I guess the optimizer searching the state space keeps quite a lot of data about what has been seen and explored.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants