[CI] Try out buildkite

[alamb]

Note: migrated from original JIRA: https://issues.apache.org/jira/browse/ARROW-11140

Let's make a test to validate whether we can use buildkite on our own flows, which adds a lot of options in architectures and environments that we can test stuff on.

Goal: validate that we can use buildkite on the rust builds.

Requirements:

pipeline starts when a PR is made

result is sent back to github and users can access its logs

we can use caches (e.g. [https://github.com/danthorpe/cache-buildkite-plugin] )

we can actually run the builds

we can limit the builds to only be triggered when certain parts of the repo change (i.e. not run when only C++ code changed)

 

[alamb]

Comment from Jorge Leitão(jorgecarleitao) @ 2021-01-06T06:07:55.582+0000:

[~wesm_impala_7e40], would it be possible to add a new pipe with a single command,

{{buildkite-agent pipeline upload .buildkite/rust.yml}}

with the default github integration?

Comment from Jorge Leitão(jorgecarleitao) @ 2021-01-06T06:16:00.080+0000:

The main blocker I see atm is that buildkite does not support starting pipelines only if some part of the code changes: [https://github.com/buildkite/feedback/issues/243]

Comment from Jorge Leitão(jorgecarleitao) @ 2021-01-06T06:26:09.316+0000:

Example of the pipeline running on a mac: [https://buildkite.com/test-811/example/builds/36#c19196dc-72dd-4c3c-868e-893dd392a63a]

can't make cache working. can't make artifacts working.

Comment from Antoine Pitrou(apitrou) @ 2021-01-06T09:39:03.248+0000:

You can still adapt the {{ci/detect_changes.py}} script to limit certain builds depending on the changed files.

Comment from Jorge Leitão(jorgecarleitao) @ 2021-01-06T19:17:45.515+0000:

[~apitrou], thanks! [~kszucs] solved this in a smarter way within buildkite. I was also able to address the caching / artifacts, which unblocks this.

I have pushed the latest version to github. [~kszucs], it assumes that there are 3 queues:
 * {{docker}}, any machine that can run a docker job using the docker plugin (basically can run docker)
 * {{macos}}, macos machine with rust installed
 * {{windows}}, windows machine with rust installed

If / when we have the runners, the pipeline should reproduce the same jobs that we have on github for Rust.

I do not have strong opinions here, I just used the same setup that we have on github wrt to the runners. It looks as follows on my repo / account, where I have one (macos) agent: [https://buildkite.com/test-811/example/builds?branch=buildkite]

 

Comment from Jorge Leitão(jorgecarleitao) @ 2021-01-06T19:20:05.869+0000:

I cannot proceed further without the runners on apache account and therefore will park for now. Thanks a lot for your help!

[jorgecarleitao]

won't fix.