Improve Flux Fuzz tests reliability

[pjbgf]

Recently a change upstream have broken all builds across fluxcd.
This is broken both at upstream build time (for our project) as well as locally within our projects. Here's the output error:

+ cd /tmp/oss_fuzz-9iS27e/go-118-fuzz-build
+ go build -o /root/go/bin/go-118-fuzz-build
+ cd addimport
/root/go/src/github.com/fluxcd/image-reflector-controller/tests/fuzz/oss_fuzz_build.sh: line 37: cd: addimport: No such file or directory
+ cleanup
+ rm -rf /tmp/oss_fuzz-9iS27e
ERROR:root:Building fuzzers failed.

This is due to changes on https://github.com/AdamKorcz/go-118-fuzz-build/ which were not backwards compatible. This package is the basis of all Go support on oss-fuzz, and therefore any changes to it will affect us.

For improved reliability we shall remove any unsupported features upstream and trying to rely solely on their setup to run smoke tests.
We will still suffer from issues when upstream changes are introduced, but this should decrease their impact.

We may also have to change the way we approach fuzz testing. Once of the key changes being to avoid sharing code amongst fuzz and non-fuzz tests. Although this is supported by Go fuzz, this isn't working well with oss-fuzz, and attempts to improve upstream support are currently stuck.

Outstanding Tasks

• Upstream changes
  • Add support for linking additional libs whilst compiling. (go native: Add support for linking static libs google/oss-fuzz#9063)
  • Create auto discovery script within fluxcd project, so that oss_fuzz_build.sh does not need to be duplicated across all flux projects. (fluxcd: Add auto discovery for Go Native fuzz targets google/oss-fuzz#9064)
• CI
  • Update make fuzz-smoketest to be based on Upstream base image
  • Use google/oss-fuzz/infra/cifuzz/actions/build_fuzzers and google/oss-fuzz/infra/cifuzz/actions/run_fuzzers on cifuzz
• pkg
  • build: Fix fuzz tests pkg#415
  • build: Improve fuzz tests' reliability pkg#416
  • Rely on upstream discovery/build scripts (fuzz: Use build script from upstream pkg#436)
• helm-controller
  • build: Fix cifuzz and improve fuzz tests' reliability helm-controller#565
  • Rely on upstream discovery/build scripts (fuzz: Use build script from upstream and fix fuzzers helm-controller#578)
• kustomize-controller
  • build: Fix cifuzz tests and improve fuzz tests' reliability kustomize-controller#771
  • Rely on upstream discovery/build scripts (fuzz: Use build script from upstream and fix fuzzers kustomize-controller#777)
• source-controller
  • Migrate to Go Native fuzz and improve reliability source-controller#965
  • build: Add postbuild script for fuzzing source-controller#968
  • Rely on upstream discovery/build scripts (fluxcd/source-controller@3a6d2a1)
• notification-controller
  • build: Fix cifuzz and improve fuzz tests' reliability notification-controller#446
  • Rely on upstream discovery/build scripts (fuzz: Use build script from upstream notification-controller#454)
• image-automation-controller
  • build: Fix cifuzz and improve fuzz tests' reliability image-automation-controller#462
  • build: Add postbuild script for fuzzing image-automation-controller#464
  • Rely on upstream discovery/build scripts (fluxcd/image-automation-controller@a18b51b)
• image-reflector-controller
  • build: Improve fuzz tests' reliability image-reflector-controller#329
  • Rely on upstream discovery/build scripts (fuzz: Use build script from upstream image-reflector-controller#330)

[AdamKorcz]

Sorry for missing this. I checked all Go projects in OSS-Fuzz for breakages but must have missed Flux since the build was broken since September: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=51202&q=proj%3Dfluxcd, or is the OSS-Fuzz build status reporting incorrect results?

If there is anything I can help with, please let me know.

[pjbgf]

@AdamKorcz Don't worry about it. Due to things like libgit2, Flux had a lot of custom code to run its fuzzers and at times it got out of sync with upstream. For example, our CI checks are based on local Dockerfiles. With the changes on this issue we should make our use of oss-fuzz more streamlined and completely aligned with oss-fuzz.

I have been working through some of the upstream build failures, but most recently (and what I referred to in the issue was) our internal CI builds also broke. Once we are done aligning the fuzz implementation all of them should be fixed.

[stefanprodan]

@pjbgf can this be closed?

[pjbgf]

Yes, closing it now.