Request to update “vulnz” user-agent to uniquely identify versions prior to 6.1.0

[Chris-Turner-NIST]

Based on traffic analysis over the past week, we believe that a vast volume of traffic contributing to ongoing NVD 2.0 API stability issues is caused by this codebase. It appears as though a recent patch, resolves an error handling implementation that can lead to large volumes of queries which will continue to repeat, due to schema validation or other related issues, when processing 2.0 API responses. This is reproducible with minimal effort using DependencyCheck 9.0.10
or older.

It appears as though nearly 50% of non-throttled/non-503 traffic coming from the "vulnz" user agent consists of sources that continuously query the APIs, fail validation or processing of the response content in some way and then repeat the process. This behavior at scale can greatly impact the stability of the API services. To ensure that data consumers who are following best practices for request volume are able to utilize the API services as intended, we intend to prevent traffic that originates from the "vulnz" user-agent.

If we understand the issue properly, we request the user-agent which appears to be set here be updated to a different value Ex: “vulnz_6.1.2”. Doing so will prevent resources being allocated for these errant, repeated, API queries. Afterwards, the dependency can be updated in any other tooling (such as ODC) so users can have a legitimate path for continued API usage.

If we misunderstand the behavior, please assist in bettering our understanding.

[Chris-Turner-NIST]

@jeremylong looks like you were already on this track before I posted the Issue with #183. Please keep us in the loop for when this kind of fix will be included in updates to dependent projects.

[jeremylong]

Correct - I realized that without supplying the version in the user agent it would be difficult to block deprecated clients. I will likely make this a breaking change and force users to upgrade (both the OSV client and ODC).

[jeremylong]

@Chris-Turner-NIST I've begun the process of moving users to the version that contains the update User-Agent. I'm hoping we can delay blocking the old user agent for a few days to give people that are paying attention time to update - if they are using 10.0.0 - 10.0.1 the correct calls are being made but these clients still use the old user-agent.

Additionally, do you know why this job is failing: https://github.com/dependency-check/DependencyCheck_Builder/actions/runs/9818834272/job/27111775951#step:4:697

Is there any chance a block was put in? I know this job was causing issues as I had forgotten to update the client being used. It is currently using the client with the updated header. The reason I'm asking is that this job actually creates a datafeed that some users of dependency-check use, as opposed to hitting the API directly.

[Chris-Turner-NIST]

@jeremylong Appreciate the quick turnaround. It looks as though a good amount of users have already transitioned to the updated user-agent. We plan to implement the block around 11:00 AM EST.

Re: https://github.com/dependency-check/DependencyCheck_Builder/actions/runs/9818834272/job/27111775951#step:4:697
This looks to align with when the stability issues began. Advise waiting to see if it is from that prior to digging in further.

[Chris-Turner-NIST]

Re: https://github.com/dependency-check/DependencyCheck_Builder/actions/runs/9818834272/job/27111775951#step:4:697 This looks to align with when the stability issues began. Advise waiting to see if it is from that prior to digging in further.

Services appear to be stabilizing already. Give this another shot and let me know if things still fail.

[jeremylong]

@Chris-Turner-NIST the job is still failing with a 404. The job is using the same API key that I use from my laptop and the same commands work locally.

[Chris-Turner-NIST]

@Chris-Turner-NIST the job is still failing with a 404. The job is using the same API key that I use from my laptop and the same commands work locally.

https://github.com/dependency-check/DependencyCheck_Builder/actions/runs/9818834272/workflow#L27

Is the GHA using a version of this lib with the updated User-Agent?

[jeremylong]

The main branch is pointing to 6.1.2 which has the new version: https://github.com/dependency-check/DependencyCheck_Builder/blob/main/.github/workflows/cache.yml#L27

[jeremylong]

@Chris-Turner-NIST I did some more testing and found that when we use the following curl:

curl -H "Accept: application/json" -H "apiKey: $NVD_API_KEY" -v https://services.nvd.nist.gov/rest/json/cves/2.0?lastModStartDate=2024-07-06T10%3A15%3A03Z&lastModEndDate=2024-11-03T10%3A15%3A03Z&resultsPerPage=2000&startIndex=0

With a valid NVD_API_KEY - we get:

< HTTP/1.1 404 
< message: Both lastModStartDate and lastModEndDate are required when either is present.

Both lastModStartDate and lastModEndDate are provided. Is the date format wrong?

lastModStartDate=2024-07-06T10%3A15%3A03Z
decoded is
lastModStartDate=2024-07-06T10:15:03Z

What is absolutely nuts to me is that the exact same curl from my MacBook works. Yet running it from a GitHub Action and it fails with the error about start/end dates.

[jeremylong]

@Chris-Turner-NIST I'm an idiot and forgot to quote the URL in the Action. The curl command works - yet vulnz is failing on the same call from the GitHub Action - but the exact same usage of vulnz works locally on my MacBook.

https://github.com/dependency-check/DependencyCheck_Builder/actions/runs/9855956400/job/27211921823

[jeremylong]

Still researching - but it does appear to be an issue with the API key. Adding additional error reporting now.

[jeremylong]

@Chris-Turner-NIST resolved the issue with my github action. I'll close this issue now as we have improved the user-agent header.

If there is anything we can do to help - let me know.

[Chris-Turner-NIST]

@Chris-Turner-NIST resolved the issue with my github action. I'll close this issue now as we have improved the user-agent header.

@jeremylong Glad you were able to figure it out. If there is something on our end that is/was causing the challenges, please let us know.

If there is anything we can do to help - let me know.

Things appear to have stabilized 👍