Alternative to this project bot

[aptalca]

We at Linuxserver.io started using this project a few years back and we desperately needed it. We have hundreds of repos for docker images and some apps that we publish and maintain and the support requests were piling up at the time, impossible for us to stay on top of. This project was a life saver.

In the beginning, it was working about 95% of the time for adding new issues to the project board, which was good enough. Moving between columns was fairly reliable (unless someone reopened a closed issue or PR, in which case the automation card would no longer be at the bottom of the column and would stop working). We dealt with the minor issues and were quite happy about it.

However, over the last couple of years, the success rate for this bot started going down. Over the last few months, it was barely working (probably at a 5-10% success rate) and it became unacceptable.

Judging by the test issues being opened here, I'm assuming others are noticing the same behavior. It's a shame, and the author seems to have stopped all work on it (I don't see any dev activity since 2019 and no responses to issues or PRs for a long time). This is not meant to be a dig at @philschatz , I apologize in advance if I cause any offense. I get it, people lose interest, or get busy with other things, and simply move on. I'm grateful for this project existing in the first place because it saved us so much time when it was working. But the reality is, we had to come up with a different solution to our problem.

We ended up creating a new project board (v2 as Github calls it, which recently came out of beta): https://github.com/orgs/linuxserver/projects/8

For managing the issues and PRs, we created a callable (reusable) workflow that handles all the logic for interacting with the project board (adding issues and PRs and moving them around). This resuable workflow is called by simpler calling workflows in each of our repos. The reason we went for this split workflow approach is so that if and when we decide to make changes to the columns or the logic, we only need to update the reusable workflow rather than updating hundreds of workflows in other repos.

Anyhow, here's the code for our calling workflow that exists in all of our repos: https://github.com/linuxserver/github-workflows/blob/v1/.github/workflows/call-issue-pr-tracker.yml
It triggers on all issues and PRs opened, reopened, labeled and unlabeled, as well as PRs with review requests added or removed. Once triggered, all it does is to call the reusable workflow. Hopefully this workflow is simple enough that we won't have to modify anytime soon (really don't want to update hundreds of repos and triggers docker builds).

Here's the code for our reusable workflow: https://github.com/linuxserver/github-workflows/blob/v1/.github/workflows/issue-pr-tracker.yml
It uses a series of ifs to determine the correct action to be taken, and then uses the following action to interact with the project board: https://github.com/leonsteinhaeuser/project-beta-automations

In a nutshell, the actions we specified are:

• Adds all issues and PRs to the project board
• Differentiates between docker related and non-docker related issues
• Acts on invalid labeling by both commenting and moving to Insufficient Info column
• Acts on review requests and moves to column PRs Ready For Team Review
• Acts on review requests and moves to column PRs Ready For Team Review
• Acts on PR reviews submitted/edited/dismissed and moves between relevant columns
• Moves closed issues and PRs to Done

EDIT: Per comment below, we ran into race condition issues and decided to let our workflow handle all actions instead of mising and matching with the v2 project built-in workflows
We also utilize some of the built-in workflows that are part of these v2 project boards (not very flexible or customizable but work great)
- Closed issues and PRs moved to Done
- PRs where review added will be moved to PRs Approved
- Merged PRs will be moved to Done

Quick update, mixing and matching the v2 project built-in workflows and our workflow utilizing this action resulted in some race condition issues.

The built-in workflows (like moving closed issues and PRs to column Done) run near instantly. Our workflow utilizing this action gets put into the regular github workflow queues and take some time to run (usually within 30 seconds or so), definitely much slower than the built-in workflows.

The edge case we ran into is triggered by things done in quick succession, such as labeling an issue as wontfix (triggers our workflow) and then closing the issue shortly after (triggers the built-in workflow).

What happens is:

1. Our workflow gets triggered on the labeling action (takes 30s or so to run).
2. The closing action triggers the built-in workflow that immediately moves it to column Done.
3. The payload for that labeling action tells our workflow that the issue is open (because it was open at the time the label was added).
4. Our workflow determines that an open issue belongs in the Issues column and moves it out of Done.
5. We end up with a closed issue in the Issues column.

As the solution, we decided not to mix and match and instead rely on our workflow for all the actions.

I hope this gives some ideas to others who can implement a similar solution.