Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add score based test evaluation #2

Closed
orangejulius opened this issue Jul 16, 2015 · 0 comments
Closed

Add score based test evaluation #2

orangejulius opened this issue Jul 16, 2015 · 0 comments
Assignees

Comments

@orangejulius
Copy link
Member

Roughly, this evaluates tests in a more granular way than simply everything passed vs anything failed as in most unit-test style suites. See the original fuzzy tests problem statement(pelias/acceptance-tests#109) for more info.

Here are some of my notes on the actual implementation details, mostly written up to get a grasp on the low level task at hand:

  • how to score a single test
    • situation:
      • many api results (as js objects with properties) returned from query
      • many expectations specified in tests (as js objects with properties)
      • both js objects potentially contain other objects nested
      • the order of the expectations and the corresponding result returned are not related
      • the basic task is to return a score, both actual and maximum, for each expectation and in turn the entire test
      • the most common situation would assign one point to each matching element within an object
        • however, additionally, different fields should be able to be given different weights to adjust for their importance
      • some expectations, instead of listing an object with properties, simply give a string
        • this string is used to look up an actual expectation object in locations.json
        • if a matching object is found, use it instead
        • if not, the test is marked as a placeholder. the score for the test is not relevant but continuing to compute it might be ok
      • determining pass/fail for the test
        • the test passed if the score is equal to the highest possible score (all expectations match), and the test is not a placeholder
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant