Project Velocity and PR reviews

[benjamingr]

Following an internal discussion about concerns of high project velocity and recent PRs landing and then being reverted, I'm copying my response publicly here (below).

I'd like to discuss what we can do to improve the PR process this during the summit if anyone is interested.

---

Thinking a little more about this - I definitely think it's a matter of not getting enough eyes to review the code rather than bigger changes.

I think this is also very discouraging to a collaborator working on a PR because:

• Having a change you worked on (and landed) reverted because someone who hasn't looked at it now looked at it is discouraging. To be clear - reverting is fine, making mistakes is fine and we should encourage experimentation - but the fact something gets approved and then pulled is frustrating.
• Having a change you worked on mostly ignored as a PR is (probably?) our San Francisco - August 6/7 2015 #1 driver of collaborator burnout. It's very frustrating and I recall collaborators that have left the org because of it.

I think that in order to improve the # of eyes on PRs we have two strategies:

• Make reviewing more rewarding somehow (list reviewers after the author in the changelog, mention noteworthy reviewers etc).
• Ping nodejs/collaborators much less and smaller teams more - so that people don't get to that vicious "I have so many node notifications in my GitHub and I'm super busy and close to burnout - let's mark all as read" which leads to stale objections and reviews.

I think we should work towards a more organized review system where:

• Relevant people who own various parts of the code form teams (like an http team etc)
• Teams report to the TSC (or write a report) once a month about the subsystem, teams would be self organizing and hopefully meet once a month.
• Teams get pinged on PRs to subsystems (mostly the case) and we stop pinging nodejs/collaborators on many things.

Optimally, a collaborator joining would automatically join the teams related to the PRs said collaborator worked on when joining - we should also likely ping existing collaborators.

Would it make sense? It's really just broad ideas.

[Tiriel]

Lots to discuss there, and I'm definitely not the best to discuss most of it, but I'm deeply interested in hearing/reading discussions about this. Burnout is a very real problem, and I feel it's even worse in OSS since we're supposed to do this because we love it.

TL;DR: huge +1 on discussing this at Summit. I'll leave the rest to competent people.

[benjamingr]

So, here is actionable stuff that might encourage reviews (some from the post above):

• List reviewers in changelog.
• Celebrate reviewing as a form of contribution by mentioning it explicitly as a form of contribution in contributing.md - our current guide is very nice (props to the authors) - we should emphasize it as a welcome form of contribution.
• Make case studies of burnout related to lack of attention or stagnation of pull requests (I can help with that and name a few).
• Discourage pings to nodejs/collaborators in collaborator guide and encourage unsubscribing from relevant issues.
• Encourage people to join subsystem teams during the on-boarding process and establish a process for the teams meeting and filing monthly/bi-monthly reports to the TSC about subsystem status and pending issues.
• Create a more digested form of ongoing efforts so that people know better where their help is needed. Some example ongoing projects that could be 'digested' so interesting parties can help review: modules, http2, fs/promises and promise APIs.

[mafintosh]

Great idea to bring this up at the summit 👍

[Tiriel]

I'll throw in my two cents again since some of this is pretty much like things currently in discussions in the CommComm restructuring effort.

Encourage people to join subsystem teams during the on-boarding process and establish a process for the teams meeting and filing monthly/bi-monthly reports to the TSC about subsystem status and pending issues.

This is precisely the point, at least in the CommComm. We've noticed people tend to think you have to join the CommComm/contribute in the CommComm repo to be a "CommComm contributor". I think there may be a bit of the same thing at work here, and that may prove to be an org-wide problem. We may have to improve the recognition of work done in the subteams as "real contribution".

Anyway, real discussion material there 😅

[mhdawson]

+1, several good ideas suggested already and definitely a good topic for the summit.

[benjamingr]

Another thing we can suggest is letting teams decide on a tracking policy.

That way teams can take longer than 72h (over weekends) or less (fast-stack) to land PRs.

Some use cases:

• A (the) documentation team can decide to allow fast-tracking documentation changes that fix typos
• The modules team can decide to postpone landing PRs until a modules team meeting.
• The test team can decide to fast-track changes as part of a particular effort (changing assert to assert.strictEquals for example.

All such request would be part of the monthly (by monthly) interaction of the team with the TSC who'd approve/vote on them.

[ryzokuken]

A hard +1 for stronger, more independent subsystem teams. We should definitely encourage people to join any subsystem teams they're interested in working with, but IMHO we'd need to properly incentivize in order to make it more successful.

This could be easily done by giving the subsystem teams perhaps a little more autonomy (they're still very much answerable to the TSC, more on below).

This could be done by allowing the subsystem teams to have atleast a bi-weekly meeting where they triage relevant issues and discuss PRs. Only members of the concerned subsystem team would land a given PR, making sure it aligns well with the current position of the team. Each team would send a single member to represent them in TSC meetings (we could rotate people or elect them, may vary from team to team) who would report the TSC on the current status of the subsystem and take directions in case they need some.

How does that sound?

[mcollina]

Considering that this would be a short presentation and a workshop at the summit, it would be fantastic if in the presentation you sum up the current pain points and the goals for the workshop session. I think it would be very nice if at the end of it we can come up with some process recommendation for the TSC to adopt and iterate on.

[apapirovski]

This could be done by allowing the subsystem teams to have atleast a bi-weekly meeting where they triage relevant issues and discuss PRs. Only members of the concerned subsystem team would land a given PR, making sure it aligns well with the current position of the team. Each team would send a single member to represent them in TSC meetings (we could rotate people or elect them, may vary from team to team) who would report the TSC on the current status of the subsystem and take directions in case they need some.

I'm strongly -1 on this approach. It will just add an extra thing to keep track of on top of the many PRs, and create an in-group of those who can afford to attend 3+ meetings every two weeks — adding to many collaborators already busy schedules. As someone that contributes to 3 teams regularly and another 2-3 fairly often, I would just have to quit any paid work and do open source full time.

[Trott]

I agree with @apapirovski above. More generally, when trying to address any issues (not just this one) at the summit, it would be best to avoid if at all possible:

• more procedures (we already have more than anyone bothers to keep track of and many already get ignored)
• more meetings (anything that can be done asynchronously should be done asynchronously--yes, I acknowledge the irony of saying this in an issue tracker devoted to a synchronous meeting of worldwide participants--the twice-a-year face-to-face meeting is important but we should still try to do as much as possible outside of such events)
• more in-groups (which is not to say more groups is a problem, and differentiating in-groups from groups that aren't in-groups can be tricky, so this is unfortunately subjective)

Now that I've made my "don't" list, here's my "do" list of things to try to do whenever feasible:

• streamlining, clarifying, or eliminating procedures
• leaning more heavily on widely-adopted tools for collaboration like GitHub and Google Docs and less heavily on tools specific to our project that only a few people understand
• decreasing reliance on small groups of people--the fact that Important Person X didn't see a pull request shouldn't mean that it can't land or that it is at high risk of getting reverted if it does land
• automation

These are guidelines and not hard rules. (For example, I'm much more open to new procedures if we are emulating another project that has already solved a problem we are trying to solve.)

[Trott]

Possible things that might help with this specific issue:

• Consider trying to solve the "getting from first-contribution to Collaborator" issue because our conversion/retention there is not great. (It's getting better, I believe. But there's room for improvement.)

• Figure out how to deploy the people we already have more effectively. Like, there are clearly some knowledgable folks with limited time and maybe we can talk directly to them about what we can do to make things more predictable for them. Some ideas that probably won't work but just to start: Focused groups can maybe try to have a rotation for paying attention to pull requests and the issue tracker. (Like I said, these ideas probably won't fly.) Or maybe people are available but get frustrated with something specific and we can at least identify what it is and what could change to increase their participation.

• Perhaps someone can brainstorm an automated tool that would help with notifications? Or maybe one already exists?

• We could work with GitHub. I know several of us are in the early-adopter program thing they have set up to talk about features. For this, the one thing I wish GitHub would let me do is filter mention-based notifications so that I can have direct mentions and one or two teams I'm on show up in a high-priority list, but have all other team mentions show up in a different list. (I need to be on some teams for privileges, but I don't need the notifications as much as I do for other teams.) Now, this may be something I want and no one else would find useful, but the point is that we can actually ask for these kinds of things, and we might even get them.

• Maybe there's some way to mark PRs as mostly-cosmetic (doc formatting, test refactoring)?

• Or maybe we need a more clearly-defined RFC type process for new features? (I know, I said avoid new procedures. But in this case, if there is a community that has done this well that we want to emulate, it is worth learning from them. I'm looking at you, Rust people.)

[richardlau]

We could work with GitHub.

I wish there was a more formal way to conduct and record the result of polls. The current ways seems to either be reactions (from which you can't tell if someone changes their vote after a decision has been reached) or tallying comments (which adds lots of noise to the PR/issue). I'm thinking here about approvals for fast tracking, but this could equally be useful for TSC votes.

Maybe there's some way to mark PRs as mostly-cosmetic (doc formatting, test refactoring)?

Labels?

[benjamingr]

I also prefer not adding something like a biweekly meeting as it would require a lot more engagement. I was thinking giving the # of meetings to each with teams presenting to the TSC once every two months.

I agree about automation @Trott, the idea of making teams more autonomous is to do less nodejs/collaborators pings - in general I think we should aim to greatly reduce the amount of churn and things someone has to read in order to contribute. That was the point of #64 (comment) at least.

Perhaps someone can brainstorm an automated tool that would help with notifications? Or maybe one already exists?

I don't think we're there yet - I think we should discuss and decide how many notifications people would expect. Stuff that is not urgent can be digested and pulled instead of pushed - stuff that is (like stuff stalling PRs) should be pushed and there should be an obvious way to tell the difference.

decreasing reliance on small groups of people--the fact that Important Person X didn't see a pull request shouldn't mean that it can't land or that it is at high risk of getting reverted if it does land

👍

We could work with GitHub. ...

That would help - but I think this is more of a cultural decision of when to ping who.

[vsemozhetbyt]

Can we make github-bot visit stalled PRs and cc focused teams according to labels?

[benjamingr]

@vsemozhetbyt that would trigger more pings overall - I think there we should be aiming for more focused pings rather than more pings.