Coveralls outage ?

[jrfnl]

I've just seen the same thing (not) happening in two different repos (both public).

• In both cases, the repos use parallel builds.
• Each job has a unique name.
• The coverage files for each job are uploaded successfully to Coveralls according to the coverallsapp/github-action@v2 action runner.
• After the jobs have finished, the workflows call the coverallsapp/github-action@v2 action runner again to run with: parallel-finished: true, which again succeeds, but then shows that Coveralls thinks there are 0 jobs associated with the build.
• Subsequently, Coveralls never reports the coverage % back, which is problematic for repos where the Coveralls task is a requirement for allowing PRs to be merged (protected branches).

Here are two example builds from different repos where I see the same pattern happening:

• PHPCSUtils:
  Workflow script: https://github.com/PHPCSStandards/PHPCSUtils/actions/runs/8547834390/workflow
  Workflow transcript: https://github.com/PHPCSStandards/PHPCSUtils/actions/runs/8547834390
  Coveralls URL: https://coveralls.io/builds/66705234
• PHPUnit Polyfills:
  Workflow script: https://github.com/Yoast/PHPUnit-Polyfills/actions/runs/8548419439/workflow?pr=159
  Workflow transcript: https://github.com/Yoast/PHPUnit-Polyfills/actions/runs/8548419439
  Coveralls URL: https://coveralls.io/builds/66706048

[jrfnl]

P.S.: I checked the status page before I wrote the above up and according to that, all should be fine....

[jeremybanka]

Can confirm I'm having this issue today also. Recent jobs are stuck in a "Pending Completion" state but there are no API errors or reported outages.

• Here's my build log on coveralls where the builds from today are stuck:

  https://coveralls.io/repos/418586/builds

• And here's a PR where coveralls never reported a status, despite the workflow passing and the action getting a successful response from the API:

  use pnpm as package manager jeremybanka/wayforge#1672

• The coveralls workflow in question:

  https://github.com/jeremybanka/wayforge/actions/runs/8548530841/job/23422379076?pr=1672

[elchininet]

I had the same but at the end the coverage was reported almost ten hours later:

elchininet/custom-sidebar#81 (comment)

New merge requests are stuck as yesterday:

elchininet/postcss-rtlcss#293

[korya]

We are experiencing the same problem. Any update on this?

[jrfnl]

I can confirm what @elchininet says - looks like there is a backlog queue of builds to be processed. I got the Coveralls coverage comment on one of the PRs I listed above just ten minutes ago, so a good 14 hours after the jobs were run and the coverage reports were submitted.

[jrfnl]

@afinetooth James, sorry for pinging you on this, but could you give us some insight in what is going on and what sort of timeline for resolution we'll be looking at ?

Not trying to put pressure on things, just trying to get some idea of timeline to adjust my own planning (as I won't run releases without a passing build).

[jeremybanka]

I had the same but at the end the coverage was reported almost ten hours later:

elchininet/custom-sidebar#81 (comment)

New merge requests are stuck as yesterday:

elchininet/postcss-rtlcss#293

Same. This morning everything's made it through my queue, but any new jobs are stuck pending, and presumably will be stuck for about ten hours

[elchininet]

At least they have noticed the issue already.

[elchininet]

The same, after 10 hours, the MR that I posted before got the Coveralls' response: elchininet/postcss-rtlcss#293

[afinetooth]

All, thank you for your reports. Apologies for the delayed response here, we've been focused on issues reported directly to support@coveralls.io. I just want to encourage everyone here to go ahead and email us at support@coveralls.io even if you have an open source (free) subscription / no subscription.

@jrfnl thanks for the heads up.

I will get an update and circle back on your individual reports here asap.

[afinetooth]

All, we believe we have resolved this issue, which you can read more about here:
https://status.coveralls.io/incidents/cj5d1148cz1b

At this point, that means that all previously unfinished builds or delayed status updates should now be completed / received. If that's not the case for you, please let me know.

It's getting late for me, so I'll leave this at the general update, but I'll check back tomorrow to address individual cases (especially if anyone's still having issues).

Thanks for your patience.

[jrfnl]

@afinetooth Thank you for these updates. I imagine it was a stressful day for you all. Get a good rest and thank you for all you do.

[elchininet]

Hi @afinetooth,

Just with the intention that you get notified as soon as possible, because the status page says "ALL SYSTEMS OPERATIONAL". I submitted a PR some minutes ago and finished OK, now a new one from 10 minutes ago, first throwed a 502 calling coveralls and second time it finished OK but the pull request is in a hanging state.

Regards

[afinetooth]

Hi @elchininet. Thanks. I'll have a look.

I was just about to look at individual cases here. I'll start with this one!

[elchininet]

Maybe something that could help you to debug. In the page of the repo, if a refresh it multiple times, sometimes it shows that the task finished and sometimes that it is pending. But it is hanging on Github.

[afinetooth]

Hi @elchininet. To follow up on this one:

• Actions Run 1
• Actions Run 2
• Coveralls Build: https://coveralls.io/builds/66748402

At first, the issue didn't appear to fit yesterday's pattern, because, aside from the 502 (which could have been a network traffic thing from our edge service provider (Cloudflare)), the issue appears to be some missing build info, namely the commit SHA for the PR HEAD, and, as a result, PR Info and a number of other details we get from the GitHub API right after creation.

From my Admin view:
https://coveralls.io/builds/66748402

I am looking deeper to find out why we couldn't obtain that info.

[...] if a refresh it multiple times, sometimes it shows that the task finished and sometimes that it is pending

Wow! That's a good catch. Never seen that before, but I'm sure that has something to do with it.

Interestingly, the build page itself consistently shows the build is "completed" as opposed to "pending," no matter how many times I refresh it:
https://coveralls.io/builds/66748402

So I'm assuming the above issue is a UI logic problem in the RECENT BUILDS section of the Repo Page.

Looking into that as well. BRB.

[elchininet]

the issue appears to be some missing build info, namely the commit SHA for the PR HEAD, and, as a result, PR Info and a number of other details we get from the GitHub API right after creation.

I can try to close the PR and open it again to see if it solves the issue.

Interestingly, the build page itself consistently shows the build is "completed" as opposed to "pending," no matter how many times I refresh it:

In my case it still shows the status in a random way. And it appears here, in the details page of the job I receive the same as you.

By the way, something that I noticed since yesterday and not related to this particular issue. The buttons to open more details in a pending task go to nowhere:

[afinetooth]

@elchininet interesting. I haven't been able to refresh the build page and see the "pending completion" state, but I see you're seeing it from the graphic you pasted, so I'll keep trying.

But, yes, in regard to that, I think the UI implementation was incomplete and didn't get a URL under that link, and is currently pulling up some default URL that's not active. I've added that to a ticket to fix asap, but you can ignore for now. It will most likely go to our doc on Parallel Builds, or to a Common Issue.

[elchininet]

Ok, no worries for the UI, it is not important.

Should I close the PR and open it again? Or maybe make a force push with another sha. Let me know if you need the PR in that state for your investigation, I don't need that change to be merged immediately.

[afinetooth]

@elchininet

the issue appears to be some missing build info, namely the commit SHA for the PR HEAD, and, as a result, PR Info and a number of other details we get from the GitHub API right after creation.

I can try to close the PR and open it again to see if it solves the issue.

Don't worry too much about it if nothing changes after a try or two. I am looking into it right now to see what we received in your report, or if there's another reason we couldn't obtain or record it. That will probably turn up the cause.

I suspect it relates to your current CI config, so question for you:

Are you sending us coverage reports for:

• push commits on your default branch and pull_request commits on your PR branches? (Minimum Recommended CI Config)? OR
• All push commits on all branches? (Ideal Recommended CI Config)?

[elchininet]

I suspect it relates to your current CI config, so question for you:

Are you sending us coverage reports for:

• push commits on your default branch and pull_request commits on your PR branches? (Minimum Recommended CI Config)? OR
• All push commits on all branches? (Ideal Recommended CI Config)?

I use the same config for all my repos. It is the first one: Minimum Recommend CI. push on master and pull_requests.

[adamdupuis]

All, thank you for your reports. Apologies for the delayed response here, we've been focused on issues reported directly to support@coveralls.io. I just want to encourage everyone here to go ahead and email us at support@coveralls.io even if you have an open source (free) subscription / no subscription.

@afinetooth Personally, I am very thankful this issue was posted here as it enabled my team member to find it and realize it's an issue with Coveralls and not just us. Otherwise we would have spent more time investigating, wasting more time (because the status page was not yet updated). Support tickets are great when it's a specific issue, but when it's a general outage, a public post is better.

[afinetooth]

I am attempting to reply to each poster here individually, but I will do it in this one comment for now.

If you have any further feedback or questions, please just reply and we'll continue in our own thread of comments.

Posters in order:

@jrfnl I checked both of your repos above, for PHPCSStandards and Yoast, and both look normal will all recent builds complete and status updates sent to GitHub. Please let me know if you're seeing any other issues. Otherwise, I'll consider your particular cases here resolved. Thanks.

@jeremybanka I reviewed your recent builds and all looks well, and confirmed that your PR has received both PR Comments and Status Update from Coveralls. Please let me know if you're seeing any other issues. Otherwise, I'll consider your particular case here resolved. Thanks.

@elchininet we discussed more above, but I have confirmed you recent builds look normal and that you've received PR comments and status updates on the two PR's you mentioned. Please let me know if you're seeing any other issues. Otherwise, I'll consider your particular case here resolved. Thanks.

@korya You didn't post a particular build or repo, but I checked all of your recent builds for all of your repos and all looks well. Please let me know if that's not the case. Otherwise, I'll consider your particular case here resolved. Thanks.

@adamdupuis Thanks for the feedback. We try our best to respond to public issues here, but often can't get to them for days or weeks due to the support load from paid accounts. That said, we do try to respond to any new public issues when there is an incident and keep all users updated there. In the meantime, I would suggest subscribing to our status page to receive updates about any further incidents. Let me know if you have any remaining issues / cases you'd like me to review, I'd be happy to. Thanks.

[elchininet]

Hi @afinetooth,
I have submitted a new PR with the same changes and that one completed successfully, even if Coveralls shows it as pending in a random way as the previous one (now the previous one is all the time resolved). I don't find any explanation for the UI and why the previous one is still stuck 🤷🏼‍♂️ but yes, take my issue as resolved.

[elchininet]

Hi @afinetooth,
Just a heads up. It started again, jobs from this afternoon are still waiting for completion, a new one from 10 minutes ago is followint the same path.
https://coveralls.io/github/elchininet/postcss-rtlcss
Regards

[afinetooth]

All, we have published a postmortem on the incident that affected your builds on Wed, Apr 3-Thu, Apr 4:
https://status.coveralls.io/incidents/cj5d1148cz1b

Let me know if you have any questions, or thoughts or ideas you'd like to share.

Thanks again for your patience.

[afinetooth]

@elchininet I am looking into your recent builds, but I don't believe your current issues relate to the incident from last week. We believe that incident was fully resolved since last Thu at 8:30pm.

Some of the the behavior does look similar, though.

The similarities are basically the "pending completion" status for some of your builds.

The differences, though, are as follows:

1. "Pending completion" state, in your case, is different - Your builds are non-parallel, so they should theoretically never be "pending completion," as that should only happen for parallel builds that have not been closed, or that have failed to complete for other reasons. I am not sure right now how your "serial" builds have wound up in this state. I am investigating that.
2. Your build times are not slow - Your recent builds, even though they are "pending completion," have all completed in 1-second to 1-minute. The incident from last week was caused by underlying backups in background job queues that were resulting in slow build times of up to several hours or more.

For these reasons I think we have a different cause, which I'm still investigating and will feed back on asap.

[elchininet]

Hi @afinetooth,

I really don‘t know if my issues are the same as the indicent of the last week, but the results are the same, my merge requests stay in a hang state waiting for Coveralls. I cannot merge those merge request and solve the open issue in my repo just because of that:

elchininet/postcss-rtlcss#299
elchininet/postcss-rtlcss#296

If you can help me with that I will appreciate it.

Note: all the rest of my repos are working as always, I am having issues only with this repo with the same configuration as the rest and which has been working in that way for years.

Regards

[jrfnl]

@afinetooth With the incident resolved and the post-mortem published, shall we close this issue ?

[afinetooth]

@jrfnl Sounds great. I've done that. 🙏

[afinetooth]

@elchininet since I believe your issue fall outside the context of the incident that was the subject of this issue, let's please move your issue to another ticket. I'll leave it up to you if you'd like to create a new ticket here or email me at support@coveralls.io, but either way, I will follow up as soon as I can and reply back (in email or to a new issue that I can create if you haven't already).

[korya]

@afinetooth Thanks for the detailed post-mortem. I confirm that this issue is fixed for us.

[elchininet]

@elchininet since I believe your issue fall outside the context of the incident that was the subject of this issue, let's please move your issue to another ticket. I'll leave it up to you if you'd like to create a new ticket here or email me at support@coveralls.io, but either way, I will follow up as soon as I can and reply back (in email or to a new issue that I can create if you haven't already).

Perfect, I‘ll create another issue and in that way we don‘t spam others that have already resolved their issues.

I already sent an email on Sunday to support@coveralls.io

Regards