test: test_update is hanging

[terriko]

Looks like test_update (which performs the NVD update) is hanging in the long tests. Not sure if that's due to data changes, network issues at NVD, old API stuff being turned off, or what. It could use some investigation.

[Shrishti1701]

@terriko Hey! I’d like to look into this issue. I’ll start by investigating potential causes, including data changes and API behavior. Let me know if there are any specific pointers or past occurrences I should check. Thanks!

[terriko]

It may be related to #4710 -- I can't remember if test_update runs everything or just NVD, but it's not impossible that it might run into memory issues as well.

[Shrishti1701]

@terriko Thanks for the update! I'll check if test_update is running into memory issues as well and see if there's any overlap with #4710. I'll report back with any findings. Let me know if you have any additional insights!

[Shrishti1701]

@terriko I checked the NVD API by running a curl request (https://services.nvd.nist.gov/rest/json/cves/2.0), and it returned a 200 OK status, so the API itself is functional.

Possible causes I’ve considered:
✅ The tool might not be handling the API response correctly.
✅ There could be network latency or a timeout issue in the update process.
✅ The way cve-bin-tool processes the NVD data may not align with recent changes in the API.
✅ If there have been API deprecations or format changes, parsing might be failing.

Since the API is responding as expected, it seems like an internal issue within cve-bin-tool. Do you have any insights or suggestions on debugging this further?

[terriko]

The NVD API is under heavy load and has a bunch of rate limiting applied, plus thanks to staffing and budget issues there I'm not going to be shocked if it's just offline randomly now or in future. So yeah, this is known to be a slightly random problem, and it's going to require some creativity to work around it.

In short:

1. The NVD API is going to hang or not respond correctly sometimes.
2. We don't want CVE-bin-tool to hang endlessly if that happens.

Some potential options:

• break up the existing test_update into mockable unit-tests so we're not reliant on the NVD API being up at all
• have a "is the NVD API responding at all?" test with an appropriate timeout so that it doesn't hang and we can quickly see form test results if that's the issue
• add a test that makes sure things don't hang if the NVD API is down (again, you'd mock the responses to make it look like it's not responding)
• fix the existing test_update test to have appropriate timeouts so it doesn't hang if something is wrong

We've already got a PR for some of the other parts of the nvd api tests, so you may want to look and see what's already covered in that and focus on timeouts as a way to avoid the hanging.

[Shrishti1701]

I went through the test failures, and it looks like the NVD API is either down or not responding correctly, which is causing issues. Based on your suggestions, I’m thinking of the following approach to improve test reliability:

1️⃣ Mock API responses instead of relying on live API calls, so tests don’t fail due to NVD API downtime.
2️⃣ Add a quick API health check before running tests—if the API is down, we can skip those tests.
3️⃣ Set timeouts to prevent tests from hanging when the API is slow.
4️⃣ Simulate API failures in tests to ensure CVE-bin-tool handles them gracefully.
5️⃣ Check existing PRs to avoid duplicate work on this issue.

Does this approach make sense, or would you suggest any modifications? Also, is there already a PR that addresses some of these issues?

[terriko]

Any of those would work, yes. You likely don't need to do all of them to address the issue.

[JigyasuRajput]

Hey! @Shrishti1701,

Here's my follow-up from IRC, I really think the suggestion already given will do the job..
(you must also check another PR for the nvd API test to prevent conflicts)

1. A quick fix would be adding timeouts to the test itself and the HTTP requests.
   something like this

@pytest.mark.timeout(60)  # Fail if test runs longer than 60 seconds
def test_update():

and for http requests..
response = requests.get(url, headers=headers, timeout=30) # 30-second timeout

2. A more long term fix would be to refactor test_update to mock the NVD API
   something like this...

def test_update(mocker):
    # Mock the API response
    mock_response = mocker.Mock()
    mock_response.json.return_value = {"mock_data": ...}
    mocker.patch("requests.get", return_value=mock_response)
    
    # Run the update logic
    ...
    assert update_was_successful

I think this should be a good start (and sorry if i messed-up the formatting)

[NeilMathew-git]

Hi! I'm interested in working on this issue as part of my GSoC 2025 preparation. Would it be okay if I took this on?

[Saksham-Sirohi]

You can find the pr for mock tests for nvd_api here #4934 Also this issue is assigned to @Shrishti1701 so, you need to ask if she has done any work

[Shrishti1701]

Hi, I am currently working on this issue and have submitted multiple PRs. There were some issues along the way, but I am resolving them.
Also you can look at the issue and if you have any suggestions do let me know.
Thanks!