Tracking spec review requests, charter review requests, spec issues, and action items

[plehegar]

The tool doesn't correlate between spec review request, charter review request, spec comments, and action items.

• For APA, it involves spec review request, charter review request, spec comments, and action items.

• For I18n, it involves spec review request, charter review request, spec comments, and action items.

• For PING, it involves spec review request, charter review request, and spec comments.

• For TAG, it involves spec review request, charter review request, spec comments. Note: the TAG rarely raises spec comments, preferred to provide comments in the spec review request issue instead.

• For Security, it's a bit of limbo so let's skip it for now. :(

This creates duplications.

A Group submits their spec review request, linking to an issue in their own repo containing the answers to a questionnaire/checklist, that issue gets linked by the tool as a spec comments as well, AND someone gets an action item assigned to look at all of this. That's a total of 3 issues for the horizontal group.

Similarly, the charter review request gets linked as a spec comment and as an action item.

We could prevent spec comments issues to be created if it's already linked from a spec review request . Or, we could ask Groups to stop creating separate issues for the questionnaire/checklist and include those in the spec review request instead.

In addition, we could prevent spec comments issues to be created if it's a charter review request.

(from @matatk)

[plehegar]

@aphillips , does this type of duplication affect the i18n wg?

[aphillips]

The duplication affects I18N from the point of view that a self-review issue (using the checklist) created by the requesting group in their own repo gets picked up in our spec comments issues.

This is actually a Good Thing. Many groups don't realize that they need to request a spec review and the pending issue created in I18N's repo alerts me that a spec review is coming or ought to be. It also gives me a chance to pre-flight their checklist and/or do other housekeeping with them. Please don't "fix" it 😄

We don't have any duplication from our spec review requests nor does the tracker track those, AFAIK.

Charter reviews don't produce any duplicate issues for us and our action items repo is completely independent.

[matatk]

Thanks @plehegar, @aphillips for all the info. We (APA) are keen to make sure we are following the process consistently with the other HR groups.

What I draw from looking at the linked repos, and @aphillips' i18n info is:

1. The tooling is being applied consistently across i18n and APA (and the other HR groups, as far as I can tell from looking at the repos you've highlighted).

2. If we (APA) want to change something, it's likely to be whether checklist issues are given the a11y-tracker label by default (this is what's causing the duplication from our perspective). But, noting @aphillips' experience, we'll think on it for a while longer.

I do have one question for you, @aphillips: For spec reviews, where do you discuss the review amongst i18n members, whilst the review is in progress? Do you create a separate action for that, or discuss it on calls, or email? It looks like you're assigning members of i18n directly to spec review threads, but I don't see if/where you have an 'internal' thread about the reviews. Or, perhaps you're finding that most spec reviews don't need a separate thread for internal discussion?

cc @JaninaSajka

[aphillips]

@matatk Thanks for the ping.

Spec reviews are assigned a "shepherd" when the review request is discussed in our teleconference (that's when we remove the pending label and move it from "review requested" to "in review". Generally, the shepherd is actually the person who does the review, although anyone can contribute issues.

Proposed issues are created in our i18n-activity repo with a label of pending. We review pending issues each week in our teleconference. If the issue is approved by the working group, the body of the issue is moved to a new issue in the spec's repo.

Most discussion of issues takes place in our teleconference, although some discussion is sometimes done in the issue itself. We tend to discourage discussion in the issue, because that can be confusing later, but we'd rather have the comments than not.

Here's a link to a recent teleconference in which we had items in "Review RADAR" and in which we discussed pending issues: https://www.w3.org/2024/02/01-i18n-minutes.html#t04

After any issues have been reviewed, the shepherd is responsible for opening the issues in the reviewed spec's repo. They are also responsible for answering many of the comments and tracking progress. When all of a spec's issues are closed, we close the review request. In practice, this is a weak spot and @xfq and I often backstop the shepherds.

Happy to discuss if any of the above is confusing, you have additional questions, or have ideas for improvements.

[matatk]

This is super helpful; thank you @aphillips.

We do not (currently) have an equivalent to your issue proposal process, which seems elegant.

How do you ratify the proposed issues, before moving them to the spec-under-review's repo? Do you find you have enough of the group's members on calls to represent consensus? Do you ever run a formal vote amongst members (e.g. via the mailing list) on proposed issues?

[r12a]

@matatk you may find the documentation i wrote of use:

• How to review specs and raise issues (for reviewers)
• Horizontal issue tracker: How to

I should also mention that we try hard to keep all discussions as tightly as possible to a minimal number of relevant GH threads.

We have an internal 'tracker' repository where we incubate an issue before communicating with the WG whose spec we reviewed. That issue also captures thoughts about progress of the issue, eg. whether and why to close the tracker, etc. If a reviewer wants to discuss what to do with an issue before it is sent to the target Working Group, or whether to close a tracker issue, etc (ie. metadata about the issue) this is done in the tracker issue, or during an i18n WG meeting.

Once we have raised an issue in the target WG's repo, any related technical discussion takes place there.

We strongly discourage side discussions in emails or via other means, since we need to be able to rebuild the thought process around an issue, often several months later after everyone has become a little hazy about why certain decisions were made. If the decision is documented in one of those 2 places, it's easier to find and reconstruct the sequence of ideas, and much easier to manage the discussion.

How do you ratify the proposed issues, before moving them to the spec-under-review's repo? Do you find you have enough of the group's members on calls to represent consensus? Do you ever run a formal vote amongst members (e.g. via the mailing list) on proposed issues?

We usually discuss briefly during the i18n WG telecon. Occasionally, if in a hurry or if we're dealing with simple comments, we may skip this - but people are notified of the issues and have a chance to put a different view, if necessary (not common). We have never had to resort to a formal vote.

hope that helps.

[matatk]

@r12a: many thanks! I used the (very clear and helpful) HOWTO when developing a CLI horizontal review tracking tool (that tool's not important right now, though :-)), and refer to it often; it's great! @plehegar pointed me to the other doc, which is also great, and your info above helps clarify things to me.

We have an internal 'tracker' repository where we incubate an issue before communicating with the WG whose spec we reviewed.

ACK; w3c/i18n-activity. APA's w3c/a11y-review, mirrors it, but we've only ever used our repo 'in reverse' - i.e. when another group wants to ask for our input on something, they add the a11y-tracker label to one of their issues (per the HOWTO process). I had not considered that we could use the repo in both directions (per your review process).

Thanks all for your time. We have one last question on this thread, for you all (cc @aphillips, @plehegar), which touches on an important topic you raised: how do you keep track of spec reviews over time?

A spec review request is for a given version of a given spec. However, the spec as a whole has a lifetime that is much longer. Some specs even change names over time.

I imagine that, within your process, you could filter the issues in w3c/i18n-activity by their s:* label. Doesn't seem like that would work across spec renames renames (which I suppose are rare), though.

We currently use our wiki for long-term tracking; there is a category for each spec, and we update the tracking page whenever we do a review: https://www.w3.org/WAI/APA/wiki/Category:Spec_Review

One example is the Clipboard API and Events spec, which has a relatively long review history: https://www.w3.org/WAI/APA/wiki/Clipboard_API_and_events

We would like to move to tracking these things, including across spec renames, more automagically, via GitHub, though.

[plehegar]

So far, I've done manual maintenance of the s:* labels to keep tracks of renaming. For example, I renamed the label s:window-placement to s:window-management recently. This way, the issues are following the new name rather than being left behind. We don't maintain the history of specification renaming in the horizontal repository. If this is a problem, I can change my habits :)