Skip to content

Composite Scoring for Frameworks

Brian Hauer edited this page Nov 19, 2019 · 2 revisions

The following is a work-in-progress feature for the TFB project. This page will be moved within this Wiki if and when the feature described here is completed.

A single composite score for frameworks

Coincident with efforts to measure and provide a score for hardware performance, we will be providing a new composite score for frameworks. The Wiki entry for TPR contains several details about how scoring works, so it is recommended that you read that first.

Below is a work-in-progress rendering of composite scores from Round 18 on Citrine.

Work in progress rendering of composite scores from Round 18

Objective

The objective of a composite score is to give a single, unified view of the relative performance of frameworks across all of our test types. While the value of this score remains to be determined, we feel it is relatively low effort to provide. We will see how the community consumes this and adjust as needed.

Limitations

Only frameworks that implement all of our test types will be included in the composite scoring chart.

Scoring

See the section on scoring within the description of TPR for details on scoring. In brief, each Round will compute new weights for each test type. These weights will be applied to each frameworks' results per test type and then summed. The resulting sum is then divided by 1,000 to yield a shorter weighted score.

Clone this wiki locally