Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fuzzer Migration Follow-ups #1903

Merged
34 changes: 16 additions & 18 deletions fuzzing/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ This directory contains files related to GitPython's suite of fuzz tests that ar
infrastructure provided by [OSS-Fuzz][oss-fuzz-repo]. This document aims to provide necessary information for working
with fuzzing in GitPython.

The latest details regarding OSS-Fuzz test status, including build logs and coverage reports, is made available
at [this link](https://introspector.oss-fuzz.com/project-profile?project=gitpython).
The latest details regarding OSS-Fuzz test status, including build logs and coverage reports, is available
on [the Open Source Fuzzing Introspection website](https://introspector.oss-fuzz.com/project-profile?project=gitpython).

## How to Contribute

Expand Down Expand Up @@ -129,47 +129,45 @@ This approach uses Docker images provided by OSS-Fuzz for building and running f
comprehensive features but requires a local clone of the OSS-Fuzz repository and sufficient disk space for Docker
containers.

#### Preparation

Set environment variables to simplify command usage:

```shell
# $SANITIZER can be either 'address' or 'undefined':
export SANITIZER=address
# specify the fuzz target without the .py extension:
export FUZZ_TARGET=fuzz_config
```

#### Build and Run
#### Build the Execution Environment

Clone the OSS-Fuzz repository and prepare the Docker environment:

```shell
git clone --depth 1 https://github.com/google/oss-fuzz.git oss-fuzz
cd oss-fuzz
python infra/helper.py build_image gitpython
python infra/helper.py build_fuzzers --sanitizer $SANITIZER gitpython
python infra/helper.py build_fuzzers --sanitizer address gitpython
```

> [!TIP]
> The `build_fuzzers` command above accepts a local file path pointing to your gitpython repository clone as the last
> The `build_fuzzers` command above accepts a local file path pointing to your GitPython repository clone as the last
> argument.
> This makes it easy to build fuzz targets you are developing locally in this repository without changing anything in
> the OSS-Fuzz repo!
> For example, if you have cloned this repository (or a fork of it) into: `~/code/GitPython`
> Then running this command would build new or modified fuzz targets using the `~/code/GitPython/fuzzing/fuzz-targets`
> directory:
> ```shell
> python infra/helper.py build_fuzzers --sanitizer $SANITIZER gitpython ~/code/GitPython
> python infra/helper.py build_fuzzers --sanitizer address gitpython ~/code/GitPython
> ```


Verify the build of your fuzzers with the optional `check_build` command:

```shell
python infra/helper.py check_build gitpython
```

#### Run a Fuzz Target

Setting an environment variable for the fuzz target argument of the execution command makes it easier to quickly select
a different target between runs:

```shell
# specify the fuzz target without the .py extension:
export FUZZ_TARGET=fuzz_config
```

Execute the desired fuzz target:

```shell
Expand Down
1 change: 0 additions & 1 deletion fuzzing/fuzz-targets/fuzz_config.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
#!/usr/bin/python3
# Copyright 2023 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
Expand Down
1 change: 0 additions & 1 deletion fuzzing/fuzz-targets/fuzz_tree.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
#!/usr/bin/python3
# Copyright 2023 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
Expand Down
8 changes: 4 additions & 4 deletions fuzzing/oss-fuzz-scripts/build.sh
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
#!/usr/bin/env bash
# shellcheck shell=bash

set -euo pipefail

python3 -m pip install .

# Directory to look in for dictionaries, options files, and seed corpa:
# Directory to look in for dictionaries, options files, and seed corpora:
SEED_DATA_DIR="$SRC/seed_data"

find "$SEED_DATA_DIR" \( -name '*_seed_corpus.zip' -o -name '*.options' -o -name '*.dict' \) \
Expand All @@ -13,7 +13,7 @@ find "$SEED_DATA_DIR" \( -name '*_seed_corpus.zip' -o -name '*.options' -o -name
-exec cp {} "$OUT" \;

# Build fuzzers in $OUT.
find "$SRC/gitpython/fuzzing" -name 'fuzz_*.py' -print0 | while IFS= read -r -d $'\0' fuzz_harness; do
find "$SRC/gitpython/fuzzing" -name 'fuzz_*.py' -print0 | while IFS= read -r -d '' fuzz_harness; do
compile_python_fuzzer "$fuzz_harness"

common_base_dictionary_filename="$SEED_DATA_DIR/__base.dict"
Expand All @@ -27,7 +27,7 @@ find "$SRC/gitpython/fuzzing" -name 'fuzz_*.py' -print0 | while IFS= read -r -d
# If a dictionary file for this fuzzer already exists and is not empty,
# we append a new line to the end of it before appending any new entries.
#
# libfuzzer will happily ignore multiple empty lines in a dictionary but crash
# LibFuzzer will happily ignore multiple empty lines in a dictionary but fail with an error
# if any single line has incorrect syntax (e.g., if we accidentally add two entries to the same line.)
# See docs for valid syntax: https://llvm.org/docs/LibFuzzer.html#id32
echo >>"$output_file"
Expand Down
13 changes: 7 additions & 6 deletions fuzzing/oss-fuzz-scripts/container-environment-bootstrap.sh
100644 → 100755
Original file line number Diff line number Diff line change
Expand Up @@ -34,23 +34,24 @@ download_and_concatenate_common_dictionaries() {
done
}

fetch_seed_corpra() {
fetch_seed_corpora() {
# Seed corpus zip files are hosted in a separate repository to avoid additional bloat in this repo.
git clone --depth 1 https://github.com/gitpython-developers/qa-assets.git qa-assets &&
rsync -avc qa-assets/gitpython/corpra/ "$SEED_DATA_DIR/" &&
rm -rf qa-assets; # Clean up the cloned repo to keep the Docker image as slim as possible.
rm -rf qa-assets # Clean up the cloned repo to keep the Docker image as slim as possible.
}

########################
# Main execution logic #
########################

fetch_seed_corpra;
fetch_seed_corpora

download_and_concatenate_common_dictionaries "$SEED_DATA_DIR/__base.dict" \
"https://raw.githubusercontent.com/google/fuzzing/master/dictionaries/utf8.dict" \
"https://raw.githubusercontent.com/google/fuzzing/master/dictionaries/url.dict";
"https://raw.githubusercontent.com/google/fuzzing/master/dictionaries/url.dict"

# The OSS-Fuzz base image has outdated dependencies by default so we upgrade them below.
python3 -m pip install --upgrade pip;
python3 -m pip install 'setuptools~=69.0' 'pyinstaller~=6.0'; # Uses the latest versions know to work at the time of this commit.
python3 -m pip install --upgrade pip
# Upgrade to the latest versions known to work at the time the below changes were introduced:
python3 -m pip install 'setuptools~=69.0' 'pyinstaller~=6.0'
Loading