-
Notifications
You must be signed in to change notification settings - Fork 270
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve testing using generated metadata #1444
Comments
We can either polish the idea here, or just start picking out the most obvious improvements and filing them as actionable bugs, examples:
and then just start adding tests once the most annoying things have been fixed |
some of the individual trickier design issues are pointed out in the code but something that maybe isn't is the interplay of
they are completely unrelated from the API perspective... yet I bet there is streamlining we could do here to make life easier for API user (and the testing code!). |
Agree that generating and serving metadata during tests is doing too many things all at once. Hard to read and debug. Test metadata and targets should be generated (ideally once) and served from their own repository/server. Cc @tedbow and friends, who I believe did some work along these lines... |
My thoughts:
|
@trishankatdatadog thanks for getting my attention. Yes we have done a few different things in https://github.com/php-tuf/php-tuf, which only a client-side implementation of the spec, related to this. TLDR: We currently have a We have tentative plans to move all our test fixtures to https://github.com/php-tuf/conformance . Ultimately it might make sense to have these fixtures out of the For example we have this proposed fixture(new PR): https://github.com/php-tuf/php-tuf/blob/tedbow-fix_terminating_2/fixtures/TUFTestFixtureTerminatingDelegation/__init__.py This is how we got to the current situation:
|
I like the idea of a test metadata/targets repo which multiple implementations can test against... |
Here are some issues with testing the updater (these apply to new and old one):
One potential solution is what I think go-tuf does: create an infrastructure the generates a vast amount of test metadata automatically, then build tests to use that metadata. This feels very appropriate for a shared conformance test suite. That approach might work for our updater tests too but I have another approach in mind as well.
This is essentially a TUF repository implementation but specific for testing the client. The Metadata is generated from scratch, is never written to disk by the test code and never needs to be loaded over the network so this side-steps a lot of the boilerplate and unneeded slow steps. Hopefully this is also leads to simpler test code but this would have to be implemented to see if that is the case. I've tested the Fetcher implementation: that works fine. Open unresolved issues that I can see are:
I think these can be figured out with a prototype. This is, I hope, a simpler problem than a repository CLI but code in https://github.com/vmware-labs/repository-editor-for-tuf/blob/main/tufrepo.py may be useful |
We're also doing this for PHP-TUF. |
I have a rough prototype of a RepositorySimulator in https://github.com/jku/tuf/commits/updater-tests-with-simulated-repo. I think it's promising. The actual testing looks like this:
in the current test infra that would have been quite painful |
cc @hosseinsia |
This is an umbrella issue for a plan I have:
The reason I want to do this is that including metadata files in the test data seems hard to scale: We should aim to test de/serialization with test data included with the test (as in #1391), and then generate the test data for the cases where this is impractical (like ngclient MetadataBundle tests).
In the end I believe the improvements will be beneficial for the future repository tools as well, but that's a longer term goal: the above list is something where we could have results in a few weeks.
For background below is what I came up with for generating full metadata for a repository and modifying it just a bit using the current Metadata API (it currently writes to files but just for debugging purposes:
to_file()
calls can be removed). There's a lot of boilerplate but also quite a few low hanging fruit that would improve the experience:(the MetadataBundle testing code is not tested yet as the branch is not up-to-date with develop)
The text was updated successfully, but these errors were encountered: