What do we do about unmaintained tracks?

[kytrinyx]

When people submit pull requests and issues to tracks that are not actively maintained it can take a very long time for anyone to respond. I keep an eye on these, but I receive a flood of exercism-related GitHub notifications, so it can take a while before I see them. This is a problem because people get disappointed and frustrated, and they're (understandably) much less likely to continue to contribute if their first contribution receives no response.

Even though the contributions to these tracks are fairly infrequent, there is a steady trickle of them, and I end up spending a significant chunk of time reviewing and responding to them.

Time that I spend reviewing/responding to those contributions is time that I don't spend improving Exercism itself. Lately I've been neglecting unmaintained tracks in order to do things like investigating the health/status of all the tracks (#97) and see if we can make progress towards a more sustainable governance model for the Exercism curriculum.

Any ideas how we should deal with them? Should we find a way to turn them off?

I don't necessarily mean turn them off on the website (people can still submit exercises and discuss solutions).

Whatever we do, we'd need a way to communicate the status of a track, both on GitHub and maybe on the site, and also a mechanism for flipping between statuses.

/cc @exercism/track-maintainers

[kotp]

I do monitor the old issues, like for Perl, opened in 2014, likely can be closed, I just can't close it. Though I could bump it, and it would bring attention to them. There are a few like that, perhaps others see, but can't really actively close.

Maybe for the track maintainers active in a language, they could "adopt an off track" or two or three, since the traffic is low there? I monitor Perl5, for example, (I can't close, and admittedly 😊 haven't bumped) among others.

And until now, I haven't thought about creating an issue to "Claim Monitor" status. But maybe creating an issue with a title like that where it is searchable, can be claimed, and then reopened if it loses its monitor/maintainer?

The problem with low communication, like opening tickets and pr's is that there is silence, and silence can be easy to overlook.

[kytrinyx]

@kotp I would be happy to give you access to Perl 5 (and thank you for monitoring! That really helps. The pings help. I'd love to find a better system, though... which is why I wrote this up.

The problem with low communication, like opening tickets and pr's is that there is silence, and silence can be easy to overlook.

Yeah, ain't that the truth!

[petertseng]

One thing that might help is to set some expectations, something like "You should expect to hear back from a track maintainer in X days. If you don't, you can inquire at _____ as to the status of the track". Perhaps it can be placed in the README or contributing doc for each track?

[petertseng]

So, interesting question. If we assume that "If you don't, you can inquire at _____ as to the status of the track" is the way things will be, then can I ask for what fills in the blank? You might think it's https://github.com/exercism/request-new-language-track , but the language track is not quite new. Or shall it be in this repo (discussions) instead?

Why am I asking this question all of a sudden? Well, what if I said that pull requests in https://github.com/exercism/xdlang/pulls have gone unseen for months (including months after the team has been @mentioned). Now, I'm willing to wait for an indefinite amount of time, but not everyone has that luxury.

I'm sorry to call one track out in particular, but often when considering things it is useful to think both generally and specifically and think about how one affects the other.

[kytrinyx]

This is a problem in many, many tracks, unfortunately.

I really would like to find a solution that we can define as a policy across the board, that:

1. Doesn't disappoint. We need to make sure that we set clear expectations. It's a terrible experience to be excited about a project, contribute, and never hear back. It's incredibly important to be responsive. Research from Mozilla shows that people who get a response (not necessarily a resolution) on their first contribution are dramatically more likely to come back and make a second contribution.
2. Empowers. People should be encouraged to take charge of tracks that are unmaintained or under-maintained, and we need to give them the tools to be successful. We need to make sure that there is a path from unmaintained to maintained. If someone is tentatively interested in taking charge, what's the playbook that they can follow to take the track from dormant to lively?

Underlying this, I want to make sure that no single person is burdened with trying to keep up with all the un(der)maintained tracks. Our decision might mean to completely deactivate them (with a clear path to reactivation), or to leave them active and set clear expectations. I don't know.

[petertseng]

This is a problem in many, many tracks

I also imagine there are other projects that face similar problems too. I suppose not all of them have the characteristic of many repositories with distinct maintainers (tracks, in our case), but I can imagine we're not alone.

Don't disappoint

I had some misgivings about my earlier proposal of "if you don't hear back in X days, ask in _____", because that places the burden on the contributor. That makes it easier on us of course, but perhaps some contributors will not have that patience... or simply miss it if we just stick it in the README!

In the other direction, if the burden is too heavy on existing members of the org, eventually someone will tire of doing it. For example, if we give some person or group of people the task of going through the tracks weekly and marking them inactive (in some way), that will become quite tiring and time-consuming.

Okay, perhaps I'll take a page out of another project's book. Since you mentioned Mozilla... There's a certain user (well, robot) https://github.com/rust-highfive whose functionality is to comment on pull requests (take a look at a few of https://github.com/rust-lang/rust/pulls ) saying you should hear from someone (it picks a random person out of a list) soon. The implementation is https://github.com/nrc/highfive.

So, an idea would to have a bot that does the same to pull requests on the org. Perhaps it could list the track maintainers (helpful since team membership isn't visible to non-org members...), and saying what to do if there is no response in X days.

Empowering

Even though I thought about it and don't think https://github.com/exercism/request-new-language-track is the right place to inquire about an undermaintained track, I imagine there are a few similarities. The resources we would suggest to a potential incoming maintainer would be at least a superset of the resources we're giving to the initial maintainers of new tracks, right? At the very least we would show https://github.com/exercism/docs/tree/master/maintaining-a-track , right?

Additional considerations I would think about:

• There may be a backlog of issues and pull requests for the track. Suggest a strategy for dealing with the backlog.
• What if the old maintainer comes back and says "WHO IS THIS IN MY TRACK?"? Even though I would hope our maintainers are good-natured people who wouldn't do that, perhaps it will happen someday. Who is prepared to resolve that when it comes? A little bit of communication beforehand will go a long way, of course.

[kytrinyx]

an idea would to have a bot that does the same to pull requests on the org [...] saying what to do if there is no response in X days.

I like this. We could use probot to write it, which would make it quite trivial to implement. A second level of improvement would be to have a playbook that people can follow to take a track from un(der)maintained to thriving. Then probot could have something about that in the comment.

Thought: un(der)maintained tracks should never have probot/stale. It would be an important part of taking on a track to triage old issues to figure out what's going on, what's important, what's not.

[I] don't think https://github.com/exercism/request-new-language-track is the right place to inquire about an undermaintained track

Yeah, I think you're right that it's not the right place, but also that we could use something similar

What if the old maintainer comes back and says "WHO IS THIS IN MY TRACK?"?

Hah. Yeah, totally a possibility. That brings up a second, related question: do we have a threshold of activity that if you drop below it, you're removed from the list of maintainers (and you no longer have commit)? We could make it trivially easy to add them back, of course.

I was chatting with @ErikSchierboom about potentially creating a probot plugin that we use to manage track maintainers using PRs instead of having to ask me to add people. The config file for that could have the list of alumni, and it would be trivially easy to give alumni commit again if they came back and were ready to participate.

If we do this, we'd need to have a set of policies around what level of activity means that you're no longer considered a maintainer, and probably tools to help detect it. I would want to do it in such a way that celebrates their contributions. I would not want them to feel like they're being punished for having a life outside of the project.

[Insti]

Is removing maintainers the right thing to be worrying about if the problem is not enough maintainers?

If I timeout due to being busy with real life for a it will be much easier for me to come back to maintaining a track if I don't then have to jump through extra hoops to regain my previous permissions.

[kytrinyx]

Is removing maintainers the right thing to be worrying about if the problem is not enough maintainers?

I think the question about removing commit access (or not) is inconsequential to the discussion. We could do it either way, and it would probably be fine.

The big question is what do we do with an un(der)maintained track to ensure that (a) contributors have a good experience, and (b) the track has a good chance of turning into a thriving track.

[petertseng]

do we have a threshold of activity

This could deserve its own issue. If it does get one, I will gladly move the following into it:

• Explain the problem we wish to solve by removing inactive maintainers.
  • Currently, it isn't "let new contributors know who they are", since non-org members can't see the team (but this may change if we use the PR bot that lists the maintainers)
  • I imagine it's something similar to "we want to know which tracks are unmaintained", right?
• I really wish that when someone no longer has the intent to be a maintainer anymore that person says so and leaves the organisation/team so the intent is clear.
  • I suspect there are some people who no longer have intent, but remained in the organisation, but obviously no direct evidence of such.
  • Do we just ask?
  • How to tell the difference between abandonment and temporarily busy? Is a long timeout period (many months) good enough? Notify if you are going to be temporarily busy?

---

If we assume that the pull request responder bot is the way to go... you know, it occurs to me that https://github.com/probot/owners is somewhat close to the desired effect, though we would want to change the message being posted slightly.

As I think we've observed a few times, tracks with 2+ maintainers are more lively because being solo is a bit lonely. But that's a little unfortunate, not everyone is going to be able to recommend someone to co-maintain.

Some strange idea, what if, if we notice 2+ people who have PRs on an unmaintained track, we recommend they review each other's PRs? Does that naturally segue into asking them to become maintainers?

[Insti]

Brainstorming/idea

Unmaintainers

It would be good if there was a group that was "unmaintained maintainers" (unmaintainers).

There would be a group of people who could share the responsiblitly of responding to issues/PRs on unmaintained tracks. If contributors are responded to they are more likely to make further contributions and can be encouraged to become maintainers.

These unmaintainers would:

• Get notifications about issues/PRs on unmaintained tracks.
• Respond to issues.
  Even if just to say, please submit a PR or, we've heard you, this track isn't very active, please work out the answer yourself and post it here. etc.
• Be able to merge PRs.
  Without needing to be subject experts, just assume that the submitter is well intentioned, the worst that can happen is the next person comes along and submits another PR correcting any added issues.

Why I'm not doing this already:

These things seem to me like they will be a lot of work (this may or may not be true):

• working out which tracks are unmaintained.
• getting write access to all the repositories.
• being able to unfollow tracks that become maintained.
• Don't want to be considered the maintainer of any unmaintained tracks.

Other thoughts:

It may be that the unmaintainers group would be the same people as the x-common maintainers.

I don't know if/how any of this is possible with vanilla GitHub, it might require a bot to add/remove permissions, group memberships etc.

There could be a bot that watched all language repositories and if an issue was not responded to (by a maintainer) in x days, that repository gets added to the unmaintained list.

[petertseng]

Without needing to be subject experts, just assume that the submitter is well intentioned, the worst that can happen is the next person comes along and submits another PR correcting any added issues.

Yup good idea. I imagine this will be a common thought - "I can't be an unmaintainer, I don't know the first thing about these languages!!!", and this point helps alleviate that.

Don't want to be considered the maintainer of any unmaintained tracks.

I'm not an expert in psychology or even anywhere close, but it seems this one will need work to overcome. If people approach this thinking "How will the benefits for me compare to the costs?" it is hard to understand the benefits of being an unmaintainer. People may think "Why do I need to think about contributors to unmaintained tracks? I've got enough going on with my own track(s) as it is!" To deal with that problem it would be good to explain how it leads to a healthier and better Exercism (ideas from one track can spread to others, more eyes on x-common, other reasons to think of).

[Insti]

Don't want to be considered the maintainer of any unmaintained tracks.

For me this is more about: I have 5 minutes today to quickly respond to an issue in x-random, but I don't want to have to guarantee that this will be true for a different issue tomorrow.

(Although practically it will be the thin edge of the wedge and I'll find myself getting sucked into learning some new random language AND maintaining the track anyway.)

[ErikSchierboom]

People may think "Why do I need to think about contributors to unmaintained tracks? I've got enough going on with my own track(s) as it is!"

This. We have to communicate very clearly what we do and don't expect from unmaintainers.

[petertseng]

How do those open source projects with similar structure handle this? I can't think of many off the top of my head, but looking at https://docs.travis-ci.com/user/languages/community-supported-languages/ may help.

I see that they ask for three maintainers to start. I wonder how that higher barrier affects things. But obviously that doesn't help us right now, since that is a preventative measure rather than a measure to take when a language is already in that state.

So the interesting question will be: Has Travis CI had to deal with situations where all the maintainers of a language are absent?

[kytrinyx]

Has Travis CI had to deal with situations where all the maintainers of a language are absent?

That's an interesting question. I'll reach out to them and ask.

[kytrinyx]

I got a response from one of my friends at Travis CI. They said that when they don't have maintainers they reach out to people who are active in their issue trackers. They still have languages and environments where they don't have in-house expertise and they don't have active maintainers.

[kotp]

For us, this means that we should reach out to those that are active in the issue trackers, but also to those active on the site in those tracks, I think.

[kytrinyx]

@kotp Yeah, that's an excellent point.

[kotp]

@kytrinyx this does mean though that there should be a group of people that can check those tracks even before they have 'mentor' status. Maybe have it tied to rikki repository rights or something? Not sure.

I would say exercism.io repository, but that seems like it would involve too many people that are maybe mostly in it for the design practice (an assumption, and probably bad) or perhaps x-common since these folks are more tuned to the exercises as it relates to the different tracks?

[kytrinyx]

Yeah, we need to think about the implementation for that. I have no problem giving people 'mentor' status on the tracks even if they won't use it to mentor people.

We could also potentially create an admin-like page that would give us that data.

[kotp]

Mentor status probably wouldn't be as efficient as the monitor page. That page could give the curious stats like who has been giving feedback, etc.

[tleen]

Closing in favor of overarching discussion in #191.