Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add CR exit criteria. #180

Merged
merged 1 commit into from
Oct 14, 2023
Merged

Add CR exit criteria. #180

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions spec/index.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@

// name of the
group: "rch",
crEnd: "2023-11-20",
testSuiteURI: "https://w3c.github.io/rdf-canon/tests/",
implementationReportURI: "https://w3c.github.io/rdf-canon/reports/",
xref: ["infra"],
Expand Down Expand Up @@ -151,6 +152,11 @@
normatively in this specification with the expectation
that a future update to this specification will replace those with normative
references to updated RDF specifications.</p>

<p>To successfully advance to Proposed Recommendation,
each test in the <a href="https://w3c.github.io/rdf-canon/tests/">test suite</a>
MUST be passed by at least three independent implementations.
Two independent implementations MUST pass all tests.</p>
Comment on lines +157 to +159
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
each test in the <a href="https://w3c.github.io/rdf-canon/tests/">test suite</a>
MUST be passed by at least three independent implementations.
Two independent implementations MUST pass all tests.</p>
there MUST be three independent implementations of every feature defined in the specification.
Two independent implementations MUST pass all tests in the <a href="https://w3c.github.io/rdf-canon/tests/">test suite</a>.</p>

I felt that the two sentences seemed to partially overlap regarding the test passage. How about revising it as above?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the wording we agreed on at TPAC, but I wasn't sure that it was clear what "every feature defined in the specification" meant, but we do know what the tests are. In other WGs, the criteria were that each test be passed by at least two independent implementations, we've exceeded this by requiring two implementations to pass all tests. The implication to me was that each test be passed by at least three independent implementations.

I'm happy to accept your wording if people feel that the meaning of "every feature ..." is clear.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think the suggestion above provides any clarification.

A pointer to the TPAC discussion and resolution would be appreciated, as I do not recall these exit criteria.

[@gkellogg] we've exceeded this by requiring two implementations to pass all tests. The implication to me was that each test be passed by at least three independent implementations.

I don't see how requiring two implementations to pass all tests implies requiring any test, never mind each, to be passed by more than two implementations. It seems to me that requiring two implementations to pass all tests only implies requiring two implementations.

That said, I am not sure why we are requiring any implementation to pass all tests, to support all features. This is a massive step beyond previous CR->PR requirements, and I cannot think of any other CR with the same exit criteria!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the link to the resolution from TPAC: https://www.w3.org/2023/09/11-rch-minutes.html#r03

The WG recommends the CR exit criteria be: 1. Three independent implementations of all features, 2. Two independent implementations that each pass all tests

Copy link
Member

@iherman iherman Oct 14, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am in favour of the original formulation of @gkellogg: "feature" is undefined, but what the individual tests are is clear.

As for

That said, I am not sure why we are requiring any implementation to pass all tests, to support all features. This is a massive step beyond previous CR->PR requirements, and I cannot think of any other CR with the same exit criteria!

Yes, it is a massive step beyond the previous CR-s, but I think if we can fulfill it (and we are fulfilling it already) it gives also a more-than-usual credibility for the value of the algorithm. Because we are talking about a security related algorithm and its implementations, this type of credibility is never too strong...

(Full disclosure: this was my idea at TPAC, ie, I proposed it during the discussion.)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gkellogg Thanks, I understand what you intended with your original phrasing. I withdraw my suggestion.

As for "massiveness", while the condition might indeed be a bit strong, given the implementations that have been presented so far, I believe it's within an acceptable range.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we are fulfilling it already

OK. Somehow I had it in my head that it was fulfilled for the old version's features/tests but not necessarily for the new version's. I have no argument with it, as things actually stand.

</section>

<section id="introduction" class="informative">
Expand Down
Loading