Increase the number of retries (1->3)

[Geod24]

CC @WalterBright

[dlang-bot]

Thanks for your pull request, @Geod24!

[WalterBright]

@Geod24 I hope this won't cause a retry if a test suite failure is actually caused by a bug and not a networking failure?

[Geod24]

@WalterBright : That's the downside of it - it will.

[WalterBright]

The obvious question - can we get a proper fix?

[PetarKirov]

Given that for a human it can be difficult to decide whether a bug is Heisenbug or not, what would be the algorithm to determine that automatically?
"Agent lost" and OOM should ideally be handled by BuildKite, but for other things it's hard to say.

[MartinNowak]

Fine for me, the bill for those runners is fairly small. Did lower it to 1 in the past since many PR problems were not intermittent, but some are and human time is quite valuable.

[Geod24]

@MartinNowak : Perhaps you could take a look at https://github.com/dlang/ci/blob/master/buildkite/Dockerfile so contributors could run an agent as well ? I have a few servers that I would gladly use as permanent runners.

[WalterBright]

the algorithm

All networking errors would be a great first approximation.

[PetarKirov]

All networking errors would be a great first approximation.

Obviously, yes. The question is how to determine if a failure is networking related. For example, IIRC some (all?) std.socket unit tests run on localhost, so there internet access is not a prerequisite. A build could fail because dub can't fetch a dependency, which could be either caused by code.dlang.org (and its mirrors) being down, or it could be that the project being built was looking for a non-existing version, etc. While a restart is likely to resolve the first cause, it is unlikely to help with second one.
The high-level idea is clear, but implementation not so much, especially given that we're running the test suites of third-party projects.
Also, IIRC, in the past several months, network related problems where much more rare than say a mismatch between compiler and druntime version occuring when the dlang/druntime project is build on BuildKite.

[WalterBright]

Over here dlang/dmd#12409 (comment) the failure is:

CI agent stopped responding!

Surely that's detectable.

[MartinNowak]

There seems to be almost zero benefit for a smart retry over a 3x blunt retry, won't even be noticeably faster.
I'd suggest to just stick with the approach here instead of wasting time on dealing with a huge error surface.

Over here dlang/dmd#12409 (comment) the failure is:

CI agent stopped responding!

IIRC there is a 5 min. wait-time for running jobs when downscaling agents.

ci/ansible/roles/buildkite_agent/defaults/main.yml

Line 4 in 338cfde

stop_timeout: 5min # systemd TimeoutStopSec for graceful agent shutdown

If the problem occurs often, we could bump that a bit if there are many long-running jobs.

@MartinNowak : Perhaps you could take a look at https://github.com/dlang/ci/blob/master/buildkite/Dockerfile so contributors could run an agent as well ? I have a few servers that I would gladly use as permanent runners.

What's the benefit of someone else running servers? Sounds nice in theory, but reliability on a heterogeneous infrastructure run by an uncoordinated group is likely to suffer.

Perhaps you could take a look at https://github.com/dlang/ci/blob/master/buildkite/Dockerfile so contributors could run an agent as well?

I guess a simpler dependency file might indeed help us to update the machines. Is this a real problem?
Could try to find some time when possible, but cannot promise anything.

• Bringing up the agents from images is surely more accessible than the ansible image creation (don't think easier image building tools like Packer would improve much).

• We could build and push image updates with GH and use them when booting servers.
  

• Personally not keen to run docker anywhere, but can just use systemd/podman for that part.
  Chapter 4. Running Containers as systemd Services with Podman Red Hat Enterprise Linux Atomic Host 7 | Red Hat Customer Portal

[WalterBright]

@MartinNowak thanks for the evaluation. I'll defer to your expertise in the matter!

[MartinNowak]

I guess a simpler dependency file might indeed help us to update the machines. Is this a real problem?
Could try to find some time when possible, but cannot promise anything.

Any opinion on whether this is an actual problem @Geod24?

[Geod24]

@MartinNowak : The lack of machine has definitely hit us in the past. Sometimes there are no agents running for a visible amount of time, although I don't recall any time when it was more than an hour. I wasn't overly bothered by it because I just hit the retry button but @WalterBright was.

[Geod24]

Something that is a bit more lacking is the ability for projects to control their dependencies. With the changes we're seeing in the CI ecosystem (Travis disappearing, Github CI raising) I was hoping we could leverage the Github runner to simplify our current pipeline. That could theoretically make it easier for core contributors to run agents, too.
I know that control over dependencies has prevented me for adding our projects here.

[MartinNowak]

Something that is a bit more lacking is the ability for projects to control their dependencies. With the changes we're seeing in the CI ecosystem (Travis disappearing, Github CI raising) I was hoping we could leverage the Github runner to simplify our current pipeline. That could theoretically make it easier for core contributors to run agents, too.
I know that control over dependencies has prevented me for adding our projects here.

Indeed we could rebuild the service in GitHub Actions 👍, might be more accessible for everyone, would require some additional setup time (hopefully fine). Not sure how long their free open source CI will last, I'd guess a while with MSFTs current strategy.