Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Next-generation CI/CD pipelines with RunsOn #11001

Merged
merged 98 commits into from
Dec 6, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
98 commits
Select commit Hold shift + click to select a range
1fa8c70
Remove existing GH workflows
hcho3 Oct 17, 2024
78c4d8c
First RunsOn example
hcho3 Oct 17, 2024
92fd6a2
Test custom runner config
hcho3 Oct 17, 2024
d6761cb
Try out GPU support
hcho3 Oct 18, 2024
3331cd6
Add Windows
hcho3 Oct 22, 2024
61f207d
Fix Windows build
hcho3 Oct 23, 2024
6b78a12
Custom image for Linux
hcho3 Oct 23, 2024
000be18
New Docker wrapper with S3 caching
hcho3 Oct 24, 2024
e1e3b41
Create utility script to build CI container
hcho3 Oct 24, 2024
5373276
Migrate some tasks from BuildKite
hcho3 Oct 25, 2024
fa96af1
Move more pipelines; refine stash_artifacts
hcho3 Oct 29, 2024
f387555
Simplify stash_artifacts
hcho3 Oct 30, 2024
91eee2d
Migrate Python tests
hcho3 Oct 30, 2024
26fff38
Mass renaming; Migrate R GPU pkg build + MacOS
hcho3 Oct 30, 2024
bf4b8ea
Fix dmlc/xgboost#10752
hcho3 Nov 1, 2024
80a883e
Fix clang-tidy + rpkg build
hcho3 Nov 1, 2024
d86deda
[WIP] Properly handle shaded JARs
hcho3 Nov 5, 2024
125b7e9
[CI] Pin Dask to 2024.10.0 (dmlc/xgboost#10995)
hcho3 Nov 13, 2024
f3ccc6f
Refactor JVM tests
hcho3 Nov 13, 2024
67d0cc6
[jvm-packages] resolve spark compatibility issue (#10917)
wbo4958 Oct 23, 2024
b65e2cc
Test GPU JVM packages
hcho3 Nov 14, 2024
f4d94a1
Disable the host numa virtual memory allocator for now. (#10934)
trivialfis Nov 2, 2024
e1c7e24
Run GPU tests with privilege escalation
hcho3 Nov 15, 2024
11b0427
Reboot after install driver; use proprietary driver for now
hcho3 Nov 17, 2024
49eaec1
Try removing --privileged flag
hcho3 Nov 17, 2024
d3482e1
Revert "Disable the host numa virtual memory allocator for now. (#109…
hcho3 Nov 17, 2024
130d303
Build JVM docs
hcho3 Nov 18, 2024
a45b24f
Re-org test scripts
hcho3 Nov 18, 2024
6dc0df2
More reorg
hcho3 Nov 18, 2024
d1d82bc
Disable the host numa virtual memory allocator for now. (#10934)
trivialfis Nov 2, 2024
ad75358
Fix run flags to Python tests
hcho3 Nov 18, 2024
06d060b
Separate workflow for JVM packages
hcho3 Nov 19, 2024
588dd67
Rename workflow files
hcho3 Nov 19, 2024
30c1470
Merge remote-tracking branch 'upstream/master'
hcho3 Nov 19, 2024
cb8f63f
Consolidate workflow defs
hcho3 Nov 19, 2024
9f9db2b
Various fixes
hcho3 Nov 19, 2024
dce4f7b
Disable dependabot for now
hcho3 Nov 19, 2024
dfd5624
Fixes
hcho3 Nov 19, 2024
0ee55c2
Don't upload artifacts from pull requests
hcho3 Nov 19, 2024
90594ff
Merge branch 'master' into xgboost-ci-ng
hcho3 Nov 19, 2024
cb55d7a
Fix
hcho3 Nov 19, 2024
377b3e7
Merge remote-tracking branch 'upstream/master'
hcho3 Nov 19, 2024
6641f7d
Fix merge conflict
hcho3 Nov 19, 2024
0727cf5
Fix
hcho3 Nov 19, 2024
3c2e5c8
Remove unneeded matrix; set max-parallel
hcho3 Nov 19, 2024
32f7406
Formatting fixes
hcho3 Nov 20, 2024
e148e29
Fix update_rapids.sh
hcho3 Nov 20, 2024
d8abb3c
Add JVM deploy
hcho3 Nov 20, 2024
9c43544
Move stash-artifacts
hcho3 Nov 20, 2024
2c2d47d
Allow most pipeline scripts to be run locally
hcho3 Nov 20, 2024
f30a0c3
Only upload if branch
hcho3 Nov 20, 2024
4162531
Troubleshoot hanging matrix jobs
hcho3 Nov 20, 2024
db5b53d
Merge remote-tracking branch 'upstream/master'
hcho3 Nov 20, 2024
2d37469
Use major version of actions
hcho3 Nov 20, 2024
eef98c7
Move all commands to separate script files
hcho3 Nov 21, 2024
bb1a152
Fix shell error
hcho3 Nov 21, 2024
5ef3454
Temporary fix
hcho3 Nov 21, 2024
5788f14
Fix deploy step
hcho3 Nov 21, 2024
ebea2b1
install bash in FreeBSD
hcho3 Nov 21, 2024
904aaee
Fix FreeBSD and SYCL
hcho3 Nov 21, 2024
0015d67
More fix
hcho3 Nov 21, 2024
b931b20
Fix
hcho3 Nov 21, 2024
b64ac1b
Fix sycl
hcho3 Nov 21, 2024
7540203
Fix
hcho3 Nov 21, 2024
a8a8c6a
Document CI
hcho3 Nov 22, 2024
d0e209c
Minor formatting fixes to doc
hcho3 Nov 22, 2024
e112219
Merge remote-tracking branch 'upstream/master'
hcho3 Nov 22, 2024
cc6dff7
Remove unneed env vars
hcho3 Nov 22, 2024
0dac170
Add example input and output for the jq script
hcho3 Nov 22, 2024
6ba8532
Temporarily build all archs for JVM
hcho3 Nov 25, 2024
9ab82e5
Fix hanging jobs
hcho3 Nov 25, 2024
97f3965
Update stash-artifacts.sh script
hcho3 Nov 25, 2024
331dab4
Fix syntax error
hcho3 Nov 25, 2024
7eb89f3
Don't use max-parallel for RunsOn runners
hcho3 Nov 25, 2024
ab4070a
stash-artifacts.ps1: Fail if unstashing failed
hcho3 Nov 25, 2024
74c6ab9
Update doc
hcho3 Nov 25, 2024
b87ca58
Uncomment CI guards
hcho3 Nov 25, 2024
3e2d8d9
Merge remote-tracking branch 'upstream/master'
hcho3 Nov 26, 2024
54902c2
Uncomment CI guards
hcho3 Nov 26, 2024
85795d7
Add local composite action miniforge-setup
hcho3 Nov 26, 2024
8e20d66
Move custom action to dmlc/xgboost-devops, with dispatching
hcho3 Nov 26, 2024
276e0ef
Document custom actions
hcho3 Nov 26, 2024
bf8844f
Test JVM GPU pkg for Scala 2.12 and 2.13
hcho3 Nov 26, 2024
f506120
Deploy JVM: Make scala version as matrix arg
hcho3 Nov 26, 2024
d1aaa82
Typo
hcho3 Nov 26, 2024
2e2ccec
Test with dev RMM
hcho3 Nov 26, 2024
d450733
gtest shouldn't depend on RMM dev
hcho3 Nov 26, 2024
0783f38
Change interface of docker_build.sh
hcho3 Nov 26, 2024
c66ef04
Merge branch 'master' into xgboost-ci-ng
hcho3 Nov 29, 2024
63e02bd
minor formatting
hcho3 Nov 29, 2024
3f804bc
Merge branch 'master' into xgboost-ci-ng
hcho3 Dec 1, 2024
e293a51
Run with --privileged
hcho3 Dec 1, 2024
480dde4
Revert "Disable the host numa virtual memory allocator for now. (#109…
hcho3 Dec 1, 2024
f66fc0d
Merge remote-tracking branch 'upstream/master'
hcho3 Dec 2, 2024
070d23f
Update doc
hcho3 Dec 3, 2024
af6e023
Merge remote-tracking branch 'upstream/master'
hcho3 Dec 6, 2024
8517fed
Merge remote-tracking branch 'upstream/master'
hcho3 Dec 6, 2024
7117328
Merge remote-tracking branch 'upstream/master'
hcho3 Dec 6, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 0 additions & 35 deletions .github/dependabot.yml

This file was deleted.

1 change: 0 additions & 1 deletion .github/runs-on.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,4 +34,3 @@ runners:
cpu: 32
family: ["c7i-flex", "c7i", "c7a", "c5", "c5a"]
image: windows-amd64

29 changes: 12 additions & 17 deletions .github/workflows/freebsd.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,20 +15,15 @@ jobs:
timeout-minutes: 20
name: A job to run test in FreeBSD
steps:
- uses: actions/checkout@v4
with:
submodules: 'true'
- name: Test in FreeBSD
id: test
uses: vmactions/freebsd-vm@v1
with:
usesh: true
prepare: |
pkg install -y cmake git ninja googletest

run: |
mkdir build
cd build
cmake .. -GNinja -DGOOGLE_TEST=ON
ninja -v
./testxgboost
- uses: actions/checkout@v4
hcho3 marked this conversation as resolved.
Show resolved Hide resolved
with:
submodules: 'true'
- name: Test in FreeBSD
id: test
uses: vmactions/freebsd-vm@v1
with:
usesh: true
prepare: |
pkg install -y cmake git ninja googletest bash
run: |
bash ops/pipeline/test-freebsd.sh
44 changes: 22 additions & 22 deletions .github/workflows/i386.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,25 +19,25 @@ jobs:
ports:
- 5000:5000
steps:
- uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
with:
submodules: 'true'
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3.7.1
with:
driver-opts: network=host
- name: Build and push container
uses: docker/build-push-action@v6
with:
context: .
file: tests/ci_build/Dockerfile.i386
push: true
tags: localhost:5000/xgboost/build-32bit:latest
cache-from: type=gha
cache-to: type=gha,mode=max
- name: Build XGBoost
run: |
docker run --rm -v $PWD:/workspace -w /workspace \
-e CXXFLAGS='-Wno-error=overloaded-virtual -Wno-error=maybe-uninitialized -Wno-error=redundant-move' \
localhost:5000/xgboost/build-32bit:latest \
tests/ci_build/build_via_cmake.sh
- uses: actions/checkout@v4
with:
submodules: 'true'
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
with:
driver-opts: network=host
- name: Build and push container
uses: docker/build-push-action@v6
with:
context: .
file: ops/docker/dockerfile/Dockerfile.i386
push: true
tags: localhost:5000/xgboost/build-32bit:latest
cache-from: type=gha
cache-to: type=gha,mode=max
- name: Build XGBoost
run: |
docker run --rm -v $PWD:/workspace -w /workspace \
-e CXXFLAGS='-Wno-error=overloaded-virtual -Wno-error=maybe-uninitialized -Wno-error=redundant-move' \
localhost:5000/xgboost/build-32bit:latest \
bash ops/script/build_via_cmake.sh
Loading
Loading