Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bors is/went haywire #7293

Closed
huonw opened this issue Jun 22, 2013 · 10 comments
Closed

bors is/went haywire #7293

huonw opened this issue Jun 22, 2013 · 10 comments

Comments

@huonw
Copy link
Member

huonw commented Jun 22, 2013

@thestinger and I were trying to get #7289 merged, but it bounced the first time so he rebased, and I re-reviewed; but bors didn't pick it up, so @thestinger closed the PR and reopened it as #7292 (with the same commit). This meant that bors picked up #7289 (the original and closed PR). Closing #7292 and reopening #7289 removed #7289 from the status page and added #7292, so @thestinger closed both, and moved the commit to #7274. Bors then merrily merged #7289 (it was closed) before the previous patch (#7210) had finished running, and then remerged #7210 so there were 3 separate auto builds happening concurrently (two of the same pull request).

bors-madness

(It seems there are several bors processes running, e.g. the multiple sets of comments on #7274.)

It got worse... 3 lots of the one pull request (#7274):

bors-madness-2

@huonw
Copy link
Member Author

huonw commented Jun 22, 2013

cc @graydon

@thestinger
Copy link
Contributor

bors seems to have successfully killed buildbot, so I think it will stop now....

@thestinger
Copy link
Contributor

nevermind, it keeps force-pushing single pull requests over and over and starting endless builds

cancelling the builds just makes it move onto the next pull request and repeat the same process of pushing them over and over :(

@huonw
Copy link
Member Author

huonw commented Jun 22, 2013

Currently at 7 builds.

bors-madness-4

#7227, #7230 & #7274.

@emberian
Copy link
Member

At 15 builds now.

@graydon
Copy link
Contributor

graydon commented Jun 22, 2013

My apologies. I tried to teach bors to cope with transient github API failures a little better yesterday. Clearly the cure was worse than the disease. I've reverted that change.

Unfortunately because github is so flaky on the API side, the larger the pull request queue gets, the more likely github will return a failure at some point during a single bors run. At 39 entries in the queue, it happens every couple of runs (i.e. we're only getting a "complete" bors-step run every 6 or 9 minutes).

I will try a different failure-coping strategy on monday. Meanwhile I've cancelled all builds and reverted the previous attempt in bors' logic. I'll let it run "normally" over the weekend.

Sorry.

@thestinger
Copy link
Contributor

@graydon: seems like bors is running very slowly and ignoring some @bors: retry comments (example: #7210)

I pushed a PR to try and started up the auto builders so I'll just manually merge that when they pass.

@graydon
Copy link
Contributor

graydon commented Jun 22, 2013

Nothing about the retry logic has changed. I expect the slow response is just it losing runs due to github api errors. I'd appreciate it if you just let it drain the queue some, rather than interfering with the tests. Manually combining multiple pull requests will also help.

@thestinger
Copy link
Contributor

Ah, nevermind my last comment. Seems to be fine now.

@graydon
Copy link
Contributor

graydon commented Jun 25, 2013

I believe the haywire-ness is solved for now. Closing this. Open a new one if it behaves badly.

@graydon graydon closed this as completed Jun 25, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants