Testing a New Stalebot in GitHub's Community Discussions

[queenofcorgis]

TL;DR - What is happening?

We are working to improve the quality of content across the GitHub Community Discussions. For the next 30 days we are going to perform a test using a custom action to label Question discussions that have not had any activity for 60 days as stale. The action will apply the label stale and comment to inform the original poster of the stale discussion. Sample draft copy is included at the bottom of this announcement.

📢 Feedback is requested on this initial test. Please comment on this post and let us know what you think!

What Discussions will be impacted?

• The action will label discussions that have been labeled Questions and have not had and activity in the past 60 days as stale.
• The action will post the sample copy (included below) as a comment on the discussion to alert the original poster, and request they take action to resolve the question. That automated reply should prompt users to return to add more information, select a reply as the solution, or close the question as out of date. I'll add some example copy we can use below.
• This will only be applied to discussions in the Copilot, Projects and Issues, and Accessibility categories as a part of this initial test.
• This action will not close the discussions automatically.

What is a Stalebot?

You can use GitHub Actions to comment on or close issues that have been inactive for a certain period of time.

Here is more info on this from the GitHub Docs: Closing inactive issues.

We're working with GitHub engineers to test this action in discussions!

Why are only limited categories impacted?

1. We have a hypothesis that using the action to close the loop on open and stale questions in the Community Discussions will improve the quality of content overall, but we want to test assumption before we risk disrupting the entire community discussions experience.
2. The initial test includes categories with high engagement, dedicated community managers, and a high level of involvement from GitHub engineers.
3. We are starting with an initial test with Discussions that are 60 days stale. We want to see how the age of posts impacts overall engagement and resolution rates. We track how long a discussion is open without a response and without an answer so we'll be able to track data on this test and use that to inform our next steps.
4. We want to hear from you! This is an experiment to benefit our users and the quality of content. If it doesn't work, we can roll it back. If it does, we can move forward and apply this across all categories.

What will happen after the trial period?

• The team will have an ongoing evaluation of how the stalebot impacts the response and resolution rates on exiting discussions.
• If we see positive indicators based on data that will be one indicator that this worked
• We will evaluate the feedback on this discussion. If you love it or hate it, we want to know! Please comment with ways you think we can improve the experience.
• If we deploy this broadly, we'll provide users with a hot-to guide here in the Community Discussions so you can apply a similar action to your own repo discussions 🚀

Example Stalebot Copy:

🕒 **Stale Discussion Alert** 🕒

This Discussion has been labeled as stale by an automated system for having no activity in the last 60 days. Please consider one the following actions:

1️⃣ Close as Out of Date: If the topic is no longer relevant, close the Discussion as "out of date" at the bottom of the page.

2️⃣ Provide More Information: Share additional details or context — or let the community know if you've found a solution on your own.

3️⃣ Mark a Reply as Answer: If your question has been answered by a reply, mark the most helpful reply as the solution.

Note: This stale notification will only apply to Discussions with the `Question` label. To learn more, see our recent stalebot announcement.

Thank you for helping bring this Discussion to a resolution! 💬

[ferdnyc]

@queenofcorgis Since you're showing the sample copy, presumably that's meant to invite input, so I have a suggestion:

Note: This stale notification will only apply to Discussions with the Question label. To learn more, see our recent stalebot announcement.

...The message could just link to the "recent stalebot announcement" (that'd be this discussion, #70478), instead of sending us on a scavenger hunt.

[GuiSousa2910]

yes

[edwardshagen94]

@queenofcorgis Since you're showing the sample copy, presumably that's meant to invite input, so I have a suggestion:

Note: This stale notification will only apply to Discussions with the Question label. To learn more, see our recent stalebot announcement.

...The message could just link to the "recent stalebot announcement" (that'd be this discussion, #70478), instead of sending us on a scavenger hunt.

[timmc-edx]

Looks like they've updated it to do so now.

[ant-techs]

Hi How can I talk to the creator/Dev of Zello as I have a proposition for him, if you could get in touch at [redacted] I would be grateful to talk with you, Best regards, Anton.

[queenofcorgis]

Hi @ant-techs ,

I've removed your private information from your comment as posting identifiable information is not allowed, per our Code of Conduct.

If you're asking how to use the piece of software that you found in a repository on GitHub, the best way to contact the maintainers of that software is to:

• Check the README for instructions on how to operate the software and pointers to documentation or troubleshooting info.
• Check the SUPPORT file, if one exists, for instructions on how to best contact the maintainers for support.
• Check the CONTRIBUTING guide, if one exists. Sometimes if there isn't a SUPPORT file, the CONTRIBUTING guide will give instructions on how to contact the maintainers for support.

All of these documents can be found, if they exist, in the repository where you found the software itself.

[Marcono1234]

Thanks for being so considerate by only limiting it to the "Question" category and not closing discussions directly.

The stale action (and other similar actions) are known to be quite disruptive in some cases, see actions/stale#719, when they close issues where the maintainer has simply not responded yet. Or in this case here, acting on questions (possibly even well written ones) where no one ever commented on.

As you mentioned, many of the questions here are answered by GitHub staff or at least someone from the community. But some also seem to have no answer at all (discussions with label stale and only a single comment). In that case only action 1, "Close as Out of Date", would be relevant.

Do you think that is a problem, and will you try to address that?

[ant-techs]

[ferdnyc]

@ant-techs

Then you should probably change your GitHub password, because 7 days ago your account posted a message to this discussion that started with, "Hi How can I talk to the creator/Dev of Zello as I have a proposition for him..."

[jsoref]

@queenofcorgis sadly, this is in the announcements category, because I'd really like to see the action mark this discussion as stale.

I'd like to hear if you in fact ran this action and what the results were.

I personally hate the stale bot. (That I send PRs to it is not an endorsement, it's awful software for a lousy purpose.)

There appear to be two constructive pieces of feedback here:

1. If you're running an experiment, you should link to the announcement about the experiment: https://github.com/orgs/community/discussions/70478#discussioncomment-7338056
2. A critique (similar to mine) of the feature: https://github.com/orgs/community/discussions/70478#discussioncomment-7440236

There's also a PR relating to this discussion:
#69778

One problem with bots is that they make a mess of sorting (by last modified) by bubbling up stale items. Sure, it might breathe be life into a thing, but, for a system built around searching, that isn't worth doing. If someone's searching for a thing, they'll find or not find it based on their search and the text of the item.

A better approach would be to improve the UX flow so that authors are encouraged to yes/no answers as they arrive.

[jricciardi]

Hi @jsoref! I can weigh in a little bit here now that we have made some updates and have more data:

With the exception of a few edge case frustrations, we are seeing a lot of overall user acceptance for the stalebot, especially after reducing the "stale" language in favor of "inactive" and "dormant" language.

Additionally, and the numbers change daily, about 5-8% of Discussions are brought to resolution after prompting from the bot; about 1-2% are discovered to actually be product feedback that was incorrectly labeled as a question (this is a clear source of confusion and frustration for some users and something we're paying close attention to); and otherwise most Discussions never get new responses or activity.

A better approach would be to improve the UX flow so that authors are encouraged to yes/no answers as they arrive.

This is great feedback! If we find evidence this type of flow would improve outcomes for the Community, we may escalate this idea.

[jsoref]

I've tripped on the question category thing. The bot didn't explain how to recategorize my feedback.

[jricciardi]

Currently this is something we primarily rely on Community Managers to action on, and don't give wide permissions to avoid malicious users juggling useful Discussions around.

[jsoref]

Well. I have a bunch of items where the bot is complaining. They were generally meant as product feedback, not "questions".

I just walked home from work because I'm still recovering from a concussion, so if a community manager could please fix mine, I'd appreciate it. I can't find the one that was just touched today, and my headache is getting worse.

[ferdnyc]

@jsoref

One problem with bots is that they make a mess of sorting (by last modified) by bubbling up stale items.

That's true, though the stalebot does also label the discussions it wades into, helpfully. That means there's always the possibility of adding -label:inactive to the search/filter terms, in order to exclude those items.

It's not ideal, but it can at least help mitigate some of the undesirable (or at least, debatably-desirable) effects.

[timmc-edx]

A bit of feedback: I'm here because the stalebot just replied to a question I've had open for some months about the GitHub API. It suggests three courses of action, all of which are frustrating to read:

• "Close as Out of Date": Nope, I still have the problem.
• "Provide More Information": Already gave all the info I have.
• "Mark a Reply as Answer": No one has answered, so there's no reply to mark.

The only useful action I could take would be something like "Do $X to get a GitHub employee to look at your question", but that's not one of the suggestions. :-) So the bot is just reminding me that there's a gap in GH's API that I can't do anything about, and there's no one to contact, and no one has responded... which doesn't feel great.

Anyway, I don't have a specific thing I think you should do differently (with regards to the bot at least) but I did want to share that the net effect was very annoying.

[jsoref]

I see a Product Feedback tag which seems reasonable.

[timmc-edx]

Hmm! It wasn't there when I asked. Maybe it got manually labeled? Anyway, I guess the process works somehow. :-)

[jsoref]

https://github.com/community/community/blob/main/.github/workflows/label-templated-discussions.yml
https://github.com/community/community/actions/runs/9101979677/job/25020504711#step:2:1

[jricciardi]

@timmc-edx Yep, that's applied by the Action. We love a good Action here :)

And thank you for submitting that as product feedback! 🚀

[jsoref]

This is going to happen to me a lot. I don't think that "Question" vs "Product Feedback" made such a distinction to me when I filed things, so I bet a fairly large number of my items will get this noise and result in me refiling things.

I viewed "Question" as "why is this product doing X?" or "could this product do Y better?". I imagine I'd use "Product Feedback" for "Z is broken".

I have a headache and an not going to try to do a systematic review of all of my discussion posts. -- it's possible that the method I used to create them mattered.

Today I'm using a mobile browser (where the discussion category was not as visible and easy enough to skip past), whereas for filing issues about GitHub Mobile, I tend to use GitHub Mobile, and otherwise I'll generally be using a desktop browser.

https://github.com/orgs/community/discussions/81741

[jsoref]

https://github.com/orgs/community/discussions?discussions_q=is%3Aopen+label%3AQuestion+%22Discussion+Activity+Reminder%22

[MasterInQuestion]

    https://github.com/orgs/community/discussions/39375#discussioncomment-9588019
    https://github.com/orgs/community/discussions/108088#discussioncomment-9588003

    Not contributive in some cases.

[ferdnyc]

As I commented there, your second link actually turned out to be an argument for the stalebot. Its poking of that discussion ultimately led to additional comments being posted and, from those, an answer being selected.

[MasterInQuestion]

[[
[ MasterInQuestion @ CE 2024-05-29 14:44:51 UTC:
https://github.com/orgs/community/discussions/108088#discussioncomment-9594805
    Maybe mere notifying the author shall be more appropriate, rather than posting here:
    Creating pointless noise mostly irrelevant. ]

----

[ ferdnyc @ CE 2024-05-29 22:41:47 UTC:
https://github.com/orgs/community/discussions/108088#discussioncomment-9598960
    That's a very strange argument to make:
    When the result of the stalebot's posting was, ultimately:
    That someone other than the author, posted comments:
    That ultimately resulted in an answer being selected.

    I'm not saying the stalebot made that happen.
    But it calling attention to the discussion clearly had the desired effect: notifying the author alone would not have. ]

----

[ ferdnyc @ CE 2024-05-29 22:44:31 UTC:
https://github.com/orgs/community/discussions/70478#discussioncomment-9598985
    As I commented there, your 2nd link actually turned out to be an argument for the stalebot:
    Its poking of that discussion ultimately led to additional comments being posted.
    .
    And, from those: an answer being selected. ]
]]

    And this is the selected answer..?
    https://github.com/orgs/community/discussions/108088#discussioncomment-9595059

    Whose another reply there doesn't seem to be of much use:
    Mere repetition, and somewhat misleading. (non-equivalent: how the same?)

    Who has also, in addition: spammed bunch circular inexplicable links of duplicate elsewhere.
    (e.g. right in this thread [ https://github.com/orgs/community/discussions/70478#discussioncomment-9609043 ])


    As demonstrated:
    The bot message is completely off-topic and adds no value in the long term:
    Only pointlessly polluting the thread. (and potentially mistag)

    If the instigation is what really wanted: same can be achieved without piling the verbiage.
    Less verbosity = More readability

[jsoref]

@ferdnyc: I agree with @MasterInQuestion, this wasted a considerable amount of my time as well as the questioners and anyone subscribed.

I'm only adding these admittedly fairly useless comments because the bot is asking me to do so. -- In the case of the selected answer, it wasn't technically useless as it pointed to how the Questioner themselves solved the problem (and when I suggested marking something as the solution, I really meant the Questioner's post saying they'd solved it, more than my own response) -- It's true that the other post was more valuable to the next person as it provided an explanation as to why that change was correct, but...

Worse, each of my interactions risks someone accusing me of misbehaving. Whether that's link spam, abuse, gaming, ...

I don't need/want to take any of these actions. But, I'm invested in getting my tickets addressed. -- And they won't be addressed if they're closed by a bot.

It's true that some tickets may make slight progress. But there are/have-to-be better approaches to achieve that aim.

Contributors who positively improve tickets (by either answering questions, or marking questions as answered, for which answers aren't later rejected by the original poster) could be given action-credits or something else of value within the GitHub ecosystem (I certainly could use those credits). Obviously if one went this way, abuse would have to result in people not being able to receive such rewards, but that's also fairly easy to administer.

[MasterInQuestion]

[[
    I'm only adding these admittedly fairly useless comments because the bot is asking me to do so.

    In the case of the selected answer, it wasn't technically useless: as it pointed to how the questioner themselves solved the problem.
    (and when I suggested marking something as the solution: I really meant the questioner's post saying they'd solved it, more than my own response)
    .
    It's true that the other post was more valuable to the next person:
    As it provided an explanation as to why that change was correct, but...

    Worse, each of my interactions risks someone accusing me of misbehaving.
    Whether that's link spam, abuse, gaming, ...

    I don't need/want to take any of these actions.
    But, I'm invested in getting my tickets addressed.
    And they won't be addressed if they're closed by a bot.

    It's true that some tickets may make slight progress.
    But there are (or must be) better approaches to achieve that aim.

    Contributors who positively improve tickets:
    |*| By either answering questions
    |*| Or marking questions as answered
    |*| For which answers aren't later rejected by the original poster
    ; could be given action-credits or something else of value within the GitHub ecosystem.
    (I certainly could use those credits)
    .
    Obviously if one went this way: abuse would have to result in people not being able to receive such rewards.
    But that's also fairly easy to administer.
]]
    I didn't say the selected answer is useless: merely have some doubt on it.
    (not strong enough to present an objection)

    Regardless, feel free to practice what you would like: as you feel happy about it.

    Note:
    Closed issues/discussions/pulls whatsoever doesn't mean no more accepting future actions. (i.e. locked)
    People may still freely continue on which.
    .
    The Closed/Open status mostly merely serves as some sort of tag. (for mere identification)

[jricciardi]

Hey there @adamlui - It looks like this Discussion was originally posted as a Question and then you edited to be a Bug, but it still had the Question label on it, triggering the bot. That's an edge case we'll have to think through solutions for. Thank you for flagging!

I've gone ahead and relabeled that Discussion for you and removed the inactive label 👍

[jsoref]

@queenofcorgis / @jricciardi: I've refiled a couple as a courtesy to your team (https://github.com/orgs/community/discussions/126580, https://github.com/orgs/community/discussions/126631, https://github.com/orgs/community/discussions/126632).

I'd appreciate it if you were to edit the labels for these remaining three from Question to Product Feedback:

https://github.com/orgs/community/discussions/60448
https://github.com/orgs/community/discussions/58143
https://github.com/orgs/community/discussions/39447

Sincerely,
Josh Soref (@jsoref 🤕 not recovering well from a concussion, and not appreciating unpaid busy work)

[flodolo]

Here's another example: https://github.com/orgs/community/discussions/47855

It's a question — potentially a bug for the documentation once there's an answer — but none of the options offered is suitable.

Also, not particularly useful that one of suggestions is to Close as Out of Date when the bot already closed the discussion (and I can't do any change to labels).

[timmc-edx]

The announcement reads, "This action will not close the discussions automatically." But I just had a question closed by the bot: "Closed #103026 as outdated." Was that action actually performed by a human, or is the bot now being used to close older issues?

[jsoref]

@queenofcorgis, @mgriffin, @jricciardi could one of you please update this announcement?

-This action will not close the discussions automatically.
+A new action added on April 12, 2024 will close the discussion automatically if no more comments are added after the warning.

[timmc-edx]

Ugh, thanks for the reference. Stalebots like this are legitimately one of the worst things about "maintainership" as practiced in many GitHub repositories.

[timmc-edx]

To expand on this: Any community that closes out questions as "outdated" or "old" is a community where I am less likely to bother asking questions or posting issues in the first place, because my work in making well-formed, readable, reproducible posts is just going to be thrown away in a month or two. It's extremely discouraging.

When I see a stalebot, I assume the maintainers are unwilling to say "no" to feature requests and that there's no one around capable of answering questions.

[Marcono1234]

@jsoref, it is probably more than that which is outdated:

• At this point this doesn't seem to be a "30 days test" anymore, but possibly something permanent.
• It seems to apply to more than the mentioned discussion categories (see source).
• The label is not called stale anymore but instead inactive.

Maybe it would be best if they completely updated the description of this announcement.

[MasterInQuestion]

    Validity itself, does not fade with time.
    Wisdom thousands years ago, remains wisdom.

[MasterInQuestion]

    How's the Spambot going
    https://github.com/orgs/community/discussions?discussions_q=involves:github-actions%5Bbot%5D+label:Question

[MasterInQuestion]

    https://github.com/orgs/community/discussions/54551#discussioncomment-9999380
    https://github.com/orgs/community/discussions/39375#discussioncomment-10175207
    Astounding...

    Hope the AI that powers Copilot alike does better decision.

[ferdnyc]

I only looked at the second one, and it was a good close. The GitHub community discussion forums are not your blog, and they are not required to host your rambling musings on the divine (which 0 other people had engaged with, in the entire time the discussion was open) indefinitely. There was no "discussion" happening.

Seriously, sign up for a Blogspot account and you can create a bog titled "cross-repo symlinks"¹ and post as much as you like about anything you like. Just remember to leave comments open on each post in perpetuity.

Notes

1. #NoSuchThing. (You're right that git submodules are the closest analogue, but wrong that they can or should be used with such granularity. With submodules you link to a specific commit of an entire repo, that's the only way it makes sense.)

[novium2]

Retiring this post as it is over 6 months old. If you have any questions, consider starting a new post.