[Build] Run OWASP Dependency Check as a scheduled build once per day

[lhotari]

Motivation

Monitoring for new library vulnerabilities without having an automated solution is prone to errors.

There's an enhancement request #8815 about "Automated security and update routine before every release". This PR will help address those aspects.

New library vulnerabilities can be detected earlier when there's a scheduled build once per day.

Modifications

• add suppressions for false positives
• add suppressions for known issues in distribution/server and make the check
  fail if new vulnerabilities are introduced
• run reports for distribution/offloaders, distribution/io and
  pulsar-sql/presto-distribution to get a complete report of all
  vulnerabilities
• upload report files as GitHub Actions artifact

Additional context

The current master branch contains a few vulnerabilities:

• libthrift-0.12.0.jar (pkg:maven/org.apache.thrift/libthrift@0.12.0, cpe:2.3:a:apache:thrift:0.12.0:::::::*) : CVE-2019-0205, CVE-2019-0210, CVE-2020-13949
  • issue Upgrade Thrift dependency in broker to solve CVE-2019-0210, CVE-2019-0205 and CVE-2020-13949 #9248
• vertx-core-3.5.4.jar (pkg:maven/io.vertx/vertx-core@3.5.4, cpe:2.3:a:eclipse:vert.x:3.5.4:::::::*) : CVE-2018-12541, CVE-2019-17640
  • more details in [Bookie] Bookie fails to start when httpServerEnabled=true after upgrading vertx to mitigate vulnerabilities #10295 . Waiting for vert.x 3.9.8 release.
• zookeeper-3.6.2.jar (pkg:maven/org.apache.zookeeper/zookeeper@3.6.2, cpe:2.3:a:apache:zookeeper:3.6.2:::::::*) : CVE-2021-21409
  • fixed by [Security] Upgrade Zookeeper to 3.6.3 #10852

The vulnerabilities mentioned above are suppressed in src/owasp-dependency-check-suppressions.xml file so that it's possible to run OWASP Dependency Check with -DfailBuildOnAnyVulnerability=true for failing the build if any new vulnerabilities are detected. This is the way how this PR sets a baseline and llbrary vulnerabilities can be tracked.

[lhotari]

Here's an example of the OWASP Dependency Check html reports that the schedule build will produce: https://github.com/lhotari/pulsar/actions/runs/914726687
Click on "owasp-dependency-check-reports" to download the zipped files (4 separate reports).

[eolivelli]

LGTM

we have a similar Job on Apache ZooKeeper project

+1

[merlimat]

Looks good. Is there a way to trigger an email notification to the dev@ list when the job fails?

[lhotari]

Looks good. Is there a way to trigger an email notification to the dev@ list when the job fails?

It's possible, but a bit complicated in GitHub Actions to send email. There are multiple ways to implement the sending of email from GitHub Actions. You need an email sending service in all solutions.

The simplest way might be with SendGrid free, adding the API key as a GitHub Actions secret and using a shell script where curl is used to send email via the SendGrid REST API.

Perhaps that could be done separately once we see how well the scheduled job works?
Notifications do get sent by GitHub when an action fails but I'm not exactly sure how this gets delivered in apache/pulsar.

[lhotari]

The results of the scheduled OWASP Dependency Check scans can be found here:
https://github.com/apache/pulsar/actions/workflows/ci-owasp-dependency-check.yaml

[hpvd]

since list of suppressed vulnerabilities grows over time (today: 27)
would be good before every release to check if suppressing of vulnerabilities is still reasonable
See #17068