ngclient: Improve testing (status tracker issue)

[jku]

I thought we had an issue for this but could not find one...

The ngclient (both updater and TrustedMetadataSet) testing is inadequate. At a minimum we need to:

• write tests to handle missing lines reported by coverage
• go through the existing updater tests, see if we want to use them and/or if we want to reproduce them

It probably makes sense to file individual issues for smaller pieces of this work and report the overall status back in this issue?

There's some R&D to be done here:

• since the architecture is new, we shouldn't just blindly copy the tests from old codebase. As an example I expect we can do a lot of tests with just the TrustedMetadataSet -- Updater is not needed for a large part.
• Trying generated test data (instead of test files in git) may make sense Improve testing using generated metadata #1444
• Figure out if there's collab opportunity in conformance testing with php-tuf folks (see Improve testing using generated metadata #1444 (comment))

[joshuagl]

I really like the idea of a shared test suite that different implementations can test against and know whether they behave the same way.

I also think we should have tests (perhaps full systems tests, though not if entirely unnecessary) that demonstrate the protections against the attacks TUF was designed to protect against.

[jku]

Yes! the fixture idea is interesting as well, someone needs to get familiar with it: https://github.com/php-tuf/php-tuf/

We should prioritize the more unit testy things (while we don't have the coverage), but nothing prevents looking at this while that happens...

[sechkova]

Work has already progressed with ngclient testing:

• Reaching 97% coverage and adding ngclient to the coverage report should be achieved with ngclient Updater: write tests to handling coverage report  #1499
• Improve testing using generated metadata #1444 handles trying generated test data (instead of test files in git)

However we are far from ready, after the above two are resolved, there is:

• Going through the existing updater tests, see if we want to use them and/or if we want to reproduce them
• Write system tests demonstrating protection against "attacks TUF was designed to protect against".
• Evaluate the fixture idea and a shared test suit between implementations

I am not closing this issue since it contains good discussing and links, I will update the description to mark it as a placeholder and status tracker.

[sechkova]

Closing in favour of #1579.