Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

There doesn't seem to be a way to exclude corpora from artifacts #84

Open
evverx opened this issue Jan 30, 2022 · 7 comments
Open

There doesn't seem to be a way to exclude corpora from artifacts #84

evverx opened this issue Jan 30, 2022 · 7 comments

Comments

@evverx
Copy link
Contributor

evverx commented Jan 30, 2022

From google/oss-fuzz#7186 (comment)

I think CFLite should pass UPLOAD_BUILD to build_fuzzers to make it possible to exclude those large corpora from artifacts. Without that environment variable I can't skip that step with something like

if [[ "$MERGE_WITH_OSS_FUZZ_CORPORA" == "yes" ]]; then

    # When the latest builds are uploaded by CFLite the large OSS-Fuzz corpora
    # should be excluded regardless of whether MERGE_WITH_OSS_FUZZ_CORPORA
    # is set to "yes" or not.
    [[ "$UPLOAD_BUILD" == "true" ]] && exit 0
    ....
@evverx
Copy link
Contributor Author

evverx commented Jan 30, 2022

I'm not sure why corpora are included in those artifacts in the first place. I think it would probably make sense to always remove all the "*_seed_corpus.zip" files before uploading them.

evverx added a commit to evverx/systemd that referenced this issue Jan 30, 2022
The idea behind this action is to make it possible to compare the
latest fuzz targets with PRs to figure out whether bugs are really
reproducible in PRs only. Since forks (including systemd-stable) are
usually based on the upstream repository where almost all the bugs
are fixed before releases are cut it should be safe to assume that
if CFLite finds bugs in PRs they are most likely introduced in those
PRs.

It should probably be brought back once google/clusterfuzzlite#84
is fixed.
evverx added a commit to evverx/systemd that referenced this issue Jan 30, 2022
The idea behind this action is to make it possible to compare the
latest fuzz targets with PRs to figure out whether bugs are really
reproducible in PRs only. Since forks (including systemd-stable) are
usually based on the upstream repository where almost all the bugs
are fixed before releases are cut it should be safe to assume that
if CFLite finds bugs in PRs they are most likely introduced in those
PRs.

It should probably be brought back once google/clusterfuzzlite#84
is fixed.
@jonathanmetzman
Copy link
Collaborator

I'm not sure why corpora are included in those artifacts in the first place. I think it would probably make sense to always remove all the "*_seed_corpus.zip" files before uploading them.

How much does this matter to you? I don't want to add too many smart features like this since it will add too much complexity

@evverx
Copy link
Contributor Author

evverx commented Feb 1, 2022

Those corpora take up about 400Mb (when they are compressed) and that's just too much I think. I can't upload huge artifacts like that on every commit (given that they are kept for 3 months by default).

@jonathanmetzman
Copy link
Collaborator

Those corpora take up about 400Mb (when they are compressed) and that's just too much I think. I can't upload huge artifacts like that on every commit (given that they are kept for 3 months by default).

I'm going to try to fix this retention policy issue.
Lemme send a PR deleting the seed corpora.

@evverx
Copy link
Contributor Author

evverx commented Feb 9, 2022

@jonathanmetzman on second thought given that the size of those corpora can be controlled by scripts running on PRs I don't think they should be even uploaded. It took some time and I kind of DOSed myself with the public OSS-Fuzz corpora accidentally but it's possible to just open a PR, replace "code_change" with "batch" and put giant files in "$OUT/" to somewhat speed up this process. I still have no idea why GitHub allows that with read-only tokens but it is what it is apparently.

@jonathanmetzman
Copy link
Collaborator

jonathanmetzman commented Apr 1, 2022

I think we can accomplish what you are asking for by allowing you to set FILESTORE=no_filestore (should be an easy fix) and by using #93

@evverx
Copy link
Contributor Author

evverx commented Apr 1, 2022

I can't seem to wrap my head around it. I think if it was possible to set FILESTORE=no_filestore without #93 it should do as well. I think I'd need #93 if I used the "batch" mode but I use only the "code-chage" mode and additionally am planning to turn on continuous builds.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants