feat(core): scoring data customizable #2353

PagoNxt-Trade · 2022-12-01T09:06:34Z

Checklist

Tests added / updated
Docs added / updated

Does this PR introduce a breaking change?

Yes
No

Additional context

This PR add a command line parameter to add a config file for results configuration which adds a scoring calculated and qualified with this results and a limit for passing

pablonxt · 2022-12-01T09:23:10Z

From PagoNxt Trade, we are very excited to share this functionality with the community!

P0lip · 2022-12-13T12:36:45Z

Hey!
Thanks a lot for the PR. I had time off recently, and now trying to catch up with everything. I'll make sure to have a pass on it this week.

P0lip

Looking pretty neat!

What do you think @mnaumanali94?

P0lip · 2022-12-18T19:14:15Z

packages/cli/src/commands/lint.ts

@@ -127,6 +135,11 @@ const lintCommand: CommandModule = {
          description: 'path/URL to a ruleset file',
          type: 'string',
        },
+        'scoring-config': {
+          alias: 's',


tbh this option is rather quite specific, therefore I'm not sure whether an alias for it makes that much sense.
Would just keep it without an alias for now.

P0lip · 2022-12-18T19:15:22Z

packages/cli/src/formatters/utils/getScoring.ts

+    scoringFile = path.join(process.cwd(), scoringFile);
+  }
+
+  const scoringConfig: ScoringConfig = JSON.parse(fs.readFileSync(scoringFile, 'utf-8')) as ScoringConfig;


Suggested change

const scoringConfig: ScoringConfig = JSON.parse(fs.readFileSync(scoringFile, 'utf-8')) as ScoringConfig;

const scoringConfig: ScoringConfig = JSON.parse(fs.promises.readFile(scoringFile, 'utf8'));

I'd also say we should impose some validation.
We could define a JSON Schema for the format and use Ajv for that purpose.
What do you think?

We've got your suggestion
For the validation ... We guess that could be a duplicated validation, with type defined on cli/src/formatters/types.ts should be enough to validate input file, but, if anybody thinks like you about this, we could add this

P0lip · 2022-12-18T19:16:50Z

docs/guides/2-cli.md

@@ -60,6 +61,92 @@ Here you can build a [custom ruleset](../getting-started/3-rulesets.md), or exte
 - [OpenAPI ruleset](../reference/openapi-rules.md)
 - [AsyncAPI ruleset](../reference/asyncapi-rules.md)

+## Scoring the API


@pamgoodrich do you want to have a quick look at the docs here?

P0lip · 2022-12-18T19:24:22Z

packages/cli/src/formatters/json.ts

+  let scoringText = '';
+  if (options.scoringConfig !== void 0) {
+    if (options.scoringConfig.customScoring !== undefined) {
+      spectralVersion = `${options.scoringConfig.customScoring} ${version as string}`;


I don't think including a version makes much sense.
The result does not directly depend on the version of the CLI client, but on the ruleset you use, so seems like we could just not include it?

P0lip · 2022-12-18T19:24:36Z

packages/cli/src/formatters/pretty.ts

  const cliui = require('cliui');
  let output = '\n';
+  if (options.scoringConfig !== void 0) {
+    if (options.scoringConfig.customScoring !== void 0) {
+      output += `${options.scoringConfig.customScoring}${version as string}\n`;


packages/cli/src/commands/lint.ts

P0lip · 2022-12-18T19:29:27Z

docs/guides/2-cli.md

+- scoringLetter : An object with key/value pairs with scoring letter and scoring percentage, that the result must be greater , for this letter
+- threshold : A number with minimum percentage value to provide valid the file we are checking
+- warningsSubtract : A boolean to setup if accumulate the result types to less the scoring percentage or stop counting on most critical result types
+- uniqueErrors : A boolean to setup a count with unique errors or with all of them


We should define what a unique error would be.
Something like "An error is considered unique if its code and message have not been seen yet" or something similar.

P0lip · 2022-12-18T19:29:43Z

packages/cli/src/formatters/utils/uniqueErrors.ts

@@ -0,0 +1,16 @@
+import { IRuleResult } from '@stoplight/spectral-core';
+
+export const uniqueErrors = (results: IRuleResult[]): IRuleResult[] => {


Suggested change

export const uniqueErrors = (results: IRuleResult[]): IRuleResult[] => {

export const getUniqueErrors = (results: IRuleResult[]): IRuleResult[] => {

P0lip · 2022-12-18T19:30:04Z

packages/cli/src/formatters/pretty.ts

+    if (scoring >= options.scoringConfig.threshold) {
+      output += chalk['green'].bold(`\u2716 PASSED!\n`);
+    } else {
+      output += chalk['red'].bold(`\u2716 NOT PASSED!\n`);


Suggested change

output += chalk['red'].bold(`\u2716 NOT PASSED!\n`);

output += chalk['red'].bold(`\u2716 FAILED!\n`);

P0lip · 2022-12-18T19:30:31Z

packages/cli/src/formatters/stylish.ts

+    if (scoring >= options.scoringConfig.threshold) {
+      output += chalk['green'].bold(`\u2716 PASSED!\n`);
+    } else {
+      output += chalk['red'].bold(`\u2716 NOT PASSED!\n`);


Suggested change

output += chalk['red'].bold(`\u2716 NOT PASSED!\n`);

output += chalk['red'].bold(`\u2716 FAILED!\n`);

…parameter scoring threshold not reached returns exit code 1, other returns 0 scoring data tests and readme documentation added

pablonxt · 2023-01-09T13:46:54Z

@pamgoodrich all your suggestions have been applied. Thanks!

shrutiparabgoogle · 2023-01-17T19:54:02Z

@PagoNxt-Trade This is interesting! I have some concerns about locking in a scoring mechanism, as flexibility to model different use cases and opinions is a significant component to retrieving value out of the system.

It might be helpful to get the perspective of some Apigee folks who are also working on a scoring approach via https://github.com/apigee/registry. I believe, in their case, they're allowing folks to provide their own custom CEL formulas to determine the score.

Any input @timburks, @shrutiparabgoogle, @theganyo?

This is interesting! Thanks for the mention.

I can see that the scoring approach mentioned here is making some assumptions:

that the score is always a percentage
the formula used to calculate the score is also opinionated (subtract from 100)

The approach that we have implemented in https://github.com/apigee/registry is flexible with the goal to accommodate scores for anything and not only lint results. Hence, we made use of CEL expressions so that users can define their own formula for calculating the score.

While that kind of flexibility was one of the goals for us, I can see that it can be an overkill for lint specific scoring. Couple of thoughts on the current approach:

Will scores always be percentages or is there a need to support a more flexible range? It might also help to add some validation while accepting a scoring config, so that users don't accidentally generate invalid scores (like negative percentages -2%?). I might have missed it if it’s already in place.
What is the purpose of letter scoring? (trying to understand the use case here). IIUC, it is mapping the generated percentage to a letter score, what difference does looking at a letter provide me vs looking at a percentage? Perhaps these are two different ways of conveying the same information and there can be flexibility in whether users want a lettered score or a percentage score?
The use of threshold and warningsSubtract flags is not very clear to me from the documentation, explicitly mentioning the formula in the docs might help gain clarity in the assumptions made.
If the user doesn’t provide a scoringConfig, is there a default config they can get started with?

pablonxt · 2023-01-18T10:54:40Z

Hi @shrutiparabgoogle

Thanks for your input. Here you have our answers.

Usually scoring is done starting with 100% and substracting a value depending on the issue severity (error, warning,...) this is a common practice for scoring. We think is a complete overkill to go for CELs

What can be a more flexible range? We see this as previous sentence (a percentage)

About the possible negative, we will run some tests to validate that it can't happen

The purpose of letter scoring is to show a nicer scoring than a raw number. Letter scoring comes as anything you like A+, A, B, C ... See qualys ssl tests results here: https://www.ssllabs.com/ssltest/

The number or percentage is required by the algorithm to be able to substract values

We can make letter scoring optional if you don't provide the configuration of it. So results will be always percentaje and optionally letter if you provide the letter scoring configuration.

The use of threshold and warningsSubtract flags is not very clear to me from the documentation, explicitly mentioning the formula in the docs might help gain clarity in the assumptions made.

threshold : A number with minimum percentage value to provide valid the file we are checking. Any scoring below this thresold will mark the API as a failure in the scoring.

warningsSubtract, we are going to change its name to onlySubtractHigherSeverityLevel and adaprt the documentation for more clarity.

onlySubtractHigherSeverityLevel : A boolean to decide if only the higher severity level who appears in the results for the API to analize, are subtracted from scoring or every severity level are subtracted from scoring.

See sample:

true

API with Errors and Warnings, only Errors substract from scoring
API with Warnings, Warnings substract from scoring

false

API with Errors and Warnings, Errors and Warnings substracts from scoring
API with Warnings, Warnings substract from scoring

If the user doesn’t provide a scoringConfig, is there a default config they can get started with?

If you don't provide scoringConfig there will be no scoring. As spectral does now.

Does this work better?

pablonxt · 2023-01-31T13:29:01Z

Hi .. do you have any update? @P0lip @pamgoodrich @mnaumanali94

P0lip · 2023-01-31T17:28:20Z

@mnaumanali94 ^

jharmn · 2023-03-20T20:04:03Z

docs/guides/2-cli.md

+- scoringSubtract : An object with key/value pair objects for every result level we want to subtract percentage, with the percentage to subtract from number of results on every result type
+- scoringLetter : An object with key/value pairs with scoring letter and scoring percentage, that the result must be greater, for this letter
+- threshold : A number with minimum percentage value to provide valid the file we are checking. Any scoring below this thresold will mark the API as a failure in the scoring.
+- onlySubtractHigherSeverityLevel : A boolean to decide if only the higher severity level who appears in the results for the API to analize, are subtracted from scoring or every severity level are subtracted from scoring.


analize-> analyze

pablonxt · 2023-04-17T08:36:27Z

@mnaumanali94 do you have any update? We need this PR merged

PagoNxt-Trade requested review from a team as code owners December 1, 2022 09:06

PagoNxt-Trade requested a review from pamgoodrich December 1, 2022 09:06

PagoNxt-Trade changed the title ~~Feature scoring data customizable~~ feat(scoring): scoring data customizable Dec 1, 2022

PagoNxt-Trade changed the title ~~feat(scoring): scoring data customizable~~ feat(core): scoring data customizable Dec 1, 2022

PagoNxt-Trade force-pushed the feature-AT-458-scoring_data_customizable branch 4 times, most recently from 406fe0f to 2e037f5 Compare December 1, 2022 10:21

PagoNxt-Trade marked this pull request as draft December 1, 2022 12:02

PagoNxt-Trade marked this pull request as ready for review December 1, 2022 13:23

PagoNxt-Trade force-pushed the feature-AT-458-scoring_data_customizable branch from cc4f5a9 to 35b9256 Compare December 1, 2022 13:29

PagoNxt-Trade marked this pull request as draft December 1, 2022 15:53

PagoNxt-Trade force-pushed the feature-AT-458-scoring_data_customizable branch 2 times, most recently from 5c0bc4b to 9d44008 Compare December 2, 2022 08:10

PagoNxt-Trade marked this pull request as ready for review December 2, 2022 08:13

PagoNxt-Trade force-pushed the feature-AT-458-scoring_data_customizable branch from 9d44008 to aa3cbc2 Compare December 2, 2022 08:32

P0lip force-pushed the develop branch 2 times, most recently from cf3ae99 to 761c65a Compare December 14, 2022 15:55

P0lip self-requested a review December 15, 2022 19:18

P0lip reviewed Dec 18, 2022

View reviewed changes

PagoNxt-Trade force-pushed the feature-AT-458-scoring_data_customizable branch 2 times, most recently from 47bac35 to 01004ea Compare December 20, 2022 11:07

PagoNxt-Trade added 5 commits December 21, 2022 12:43

feat(core): scoring data customizable added with config file and CLI …

fb5ec2f

…parameter scoring threshold not reached returns exit code 1, other returns 0 scoring data tests and readme documentation added

feat(core): cli new parameter alias removed

b74588c

feat(core): cli results version output removed

294ec3a

feat(core): scoring config file readed with promises

ded50ec

feat(core): added unique error specified

c086041

docs(cli): clarified info and changes applied

aeb85a4

PagoNxt-Trade force-pushed the feature-AT-458-scoring_data_customizable branch from 1b682b7 to aeb85a4 Compare January 25, 2023 09:03

docs(cli): fixed lint errors

2ab64a9

Merge branch 'develop' into feature-AT-458-scoring_data_customizable

90d024c

P0lip force-pushed the develop branch from 5c4928b to b8e51b4 Compare February 3, 2023 11:42

Merge branch 'develop' into feature-AT-458-scoring_data_customizable

3815428

jharmn reviewed Mar 20, 2023

View reviewed changes

P0lip force-pushed the develop branch 7 times, most recently from dc9d7f4 to 44c40e2 Compare May 23, 2023 22:56

P0lip force-pushed the develop branch 4 times, most recently from 02ec0d4 to 84faec8 Compare June 9, 2023 19:43

P0lip force-pushed the develop branch 3 times, most recently from 9e92f34 to 6d09915 Compare September 20, 2023 18:42

P0lip force-pushed the develop branch 3 times, most recently from dc90b7a to c22f408 Compare April 4, 2024 13:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(core): scoring data customizable #2353

feat(core): scoring data customizable #2353

PagoNxt-Trade commented Dec 1, 2022

pablonxt commented Dec 1, 2022

P0lip commented Dec 13, 2022

P0lip left a comment

P0lip Dec 18, 2022

P0lip Dec 18, 2022

PagoNxt-Trade Dec 19, 2022

P0lip Dec 18, 2022

P0lip Dec 18, 2022

P0lip Dec 18, 2022

P0lip Dec 18, 2022

P0lip Dec 18, 2022

P0lip Dec 18, 2022

P0lip Dec 18, 2022

pablonxt commented Jan 9, 2023

shrutiparabgoogle commented Jan 17, 2023

pablonxt commented Jan 18, 2023

pablonxt commented Jan 31, 2023

P0lip commented Jan 31, 2023

jharmn Mar 20, 2023

pablonxt commented Apr 17, 2023

	const scoringConfig: ScoringConfig = JSON.parse(fs.readFileSync(scoringFile, 'utf-8')) as ScoringConfig;
	const scoringConfig: ScoringConfig = JSON.parse(fs.promises.readFile(scoringFile, 'utf8'));

		@@ -0,0 +1,16 @@
		import { IRuleResult } from '@stoplight/spectral-core';

		export const uniqueErrors = (results: IRuleResult[]): IRuleResult[] => {

	export const uniqueErrors = (results: IRuleResult[]): IRuleResult[] => {
	export const getUniqueErrors = (results: IRuleResult[]): IRuleResult[] => {

	output += chalk['red'].bold(`\u2716 NOT PASSED!\n`);
	output += chalk['red'].bold(`\u2716 FAILED!\n`);

feat(core): scoring data customizable #2353

Are you sure you want to change the base?

feat(core): scoring data customizable #2353

Conversation

PagoNxt-Trade commented Dec 1, 2022

pablonxt commented Dec 1, 2022

P0lip commented Dec 13, 2022

P0lip left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pablonxt commented Jan 9, 2023

shrutiparabgoogle commented Jan 17, 2023

pablonxt commented Jan 18, 2023

pablonxt commented Jan 31, 2023

P0lip commented Jan 31, 2023

Choose a reason for hiding this comment

pablonxt commented Apr 17, 2023