Skip to content

Commit 49077c5

Browse files
CentrilCentril
authored
Rollup merge of rust-lang#66090 - pietroalbini:ci-improvements, r=alexcrichton
Misc CI improvements This PR contains some misc improvements to our CI configuration: * The environment variables for MinGW builders were greatly simplified, with just `CUSTOM_MINGW=1` to tell the install scripts to install the vendored copy. All the others (`MINGW_URL`, `MINGW_DIR`, `MINGW_ARCHIVE` and `MSYS_BITS`) are detected either from the builder name or the environment. * Collecting CPU stats and running the build were moved into scripts. * Toolstate scripts validation was previously a separate step, ran just when `IMAGE=mingw-check`. This moves the validation code inside the actual image. * Vendored copies are now fetched from https://ci-mirrors.rust-lang.org instead of directly from the bucket. r? @alexcrichton
2 parents 2cad8bb + 85132b2 commit 49077c5

20 files changed

+114
-101
lines changed

Diff for: ‎src/ci/azure-pipelines/auto.yml

+6-35
Original file line numberDiff line numberDiff line change
@@ -223,37 +223,31 @@ jobs:
223223
matrix:
224224
# 32/64 bit MSVC tests
225225
x86_64-msvc-1:
226-
MSYS_BITS: 64
227226
RUST_CONFIGURE_ARGS: --build=x86_64-pc-windows-msvc --enable-profiler
228227
SCRIPT: make ci-subset-1
229228
# FIXME(#59637)
230229
NO_DEBUG_ASSERTIONS: 1
231230
NO_LLVM_ASSERTIONS: 1
232231
x86_64-msvc-2:
233-
MSYS_BITS: 64
234232
RUST_CONFIGURE_ARGS: --build=x86_64-pc-windows-msvc --enable-profiler
235233
SCRIPT: make ci-subset-2
236234
i686-msvc-1:
237-
MSYS_BITS: 32
238235
RUST_CONFIGURE_ARGS: --build=i686-pc-windows-msvc
239236
SCRIPT: make ci-subset-1
240237
# FIXME(#59637)
241238
NO_DEBUG_ASSERTIONS: 1
242239
NO_LLVM_ASSERTIONS: 1
243240
i686-msvc-2:
244-
MSYS_BITS: 32
245241
RUST_CONFIGURE_ARGS: --build=i686-pc-windows-msvc
246242
SCRIPT: make ci-subset-2
247243
# FIXME(#59637)
248244
NO_DEBUG_ASSERTIONS: 1
249245
NO_LLVM_ASSERTIONS: 1
250246
# MSVC aux tests
251247
x86_64-msvc-aux:
252-
MSYS_BITS: 64
253248
RUST_CHECK_TARGET: check-aux EXCLUDE_CARGO=1
254249
RUST_CONFIGURE_ARGS: --build=x86_64-pc-windows-msvc
255250
x86_64-msvc-cargo:
256-
MSYS_BITS: 64
257251
SCRIPT: python x.py test src/tools/cargotest src/tools/cargo
258252
RUST_CONFIGURE_ARGS: --build=x86_64-pc-windows-msvc
259253
VCVARS_BAT: vcvars64.bat
@@ -262,10 +256,8 @@ jobs:
262256
NO_LLVM_ASSERTIONS: 1
263257
# MSVC tools tests
264258
x86_64-msvc-tools:
265-
MSYS_BITS: 64
266259
SCRIPT: src/ci/docker/x86_64-gnu-tools/checktools.sh x.py /tmp/toolstate/toolstates.json windows
267260
RUST_CONFIGURE_ARGS: --build=x86_64-pc-windows-msvc --save-toolstates=/tmp/toolstate/toolstates.json
268-
DEPLOY_TOOLSTATES_JSON: toolstates-windows.json
269261

270262
# 32/64-bit MinGW builds.
271263
#
@@ -281,43 +273,30 @@ jobs:
281273
# came from the mingw-w64 SourceForge download site. Unfortunately
282274
# SourceForge is notoriously flaky, so we mirror it on our own infrastructure.
283275
i686-mingw-1:
284-
MSYS_BITS: 32
285276
RUST_CONFIGURE_ARGS: --build=i686-pc-windows-gnu
286277
SCRIPT: make ci-mingw-subset-1
287-
MINGW_URL: https://rust-lang-ci-mirrors.s3-us-west-1.amazonaws.com/rustc
288-
MINGW_ARCHIVE: i686-6.3.0-release-posix-dwarf-rt_v5-rev2.7z
289-
MINGW_DIR: mingw32
278+
CUSTOM_MINGW: 1
290279
# FIXME(#59637)
291280
NO_DEBUG_ASSERTIONS: 1
292281
NO_LLVM_ASSERTIONS: 1
293282
i686-mingw-2:
294-
MSYS_BITS: 32
295283
RUST_CONFIGURE_ARGS: --build=i686-pc-windows-gnu
296284
SCRIPT: make ci-mingw-subset-2
297-
MINGW_URL: https://rust-lang-ci-mirrors.s3-us-west-1.amazonaws.com/rustc
298-
MINGW_ARCHIVE: i686-6.3.0-release-posix-dwarf-rt_v5-rev2.7z
299-
MINGW_DIR: mingw32
285+
CUSTOM_MINGW: 1
300286
x86_64-mingw-1:
301-
MSYS_BITS: 64
302287
SCRIPT: make ci-mingw-subset-1
303288
RUST_CONFIGURE_ARGS: --build=x86_64-pc-windows-gnu
304-
MINGW_URL: https://rust-lang-ci-mirrors.s3-us-west-1.amazonaws.com/rustc
305-
MINGW_ARCHIVE: x86_64-6.3.0-release-posix-seh-rt_v5-rev2.7z
306-
MINGW_DIR: mingw64
289+
CUSTOM_MINGW: 1
307290
# FIXME(#59637)
308291
NO_DEBUG_ASSERTIONS: 1
309292
NO_LLVM_ASSERTIONS: 1
310293
x86_64-mingw-2:
311-
MSYS_BITS: 64
312294
SCRIPT: make ci-mingw-subset-2
313295
RUST_CONFIGURE_ARGS: --build=x86_64-pc-windows-gnu
314-
MINGW_URL: https://rust-lang-ci-mirrors.s3-us-west-1.amazonaws.com/rustc
315-
MINGW_ARCHIVE: x86_64-6.3.0-release-posix-seh-rt_v5-rev2.7z
316-
MINGW_DIR: mingw64
296+
CUSTOM_MINGW: 1
317297

318298
# 32/64 bit MSVC and GNU deployment
319299
dist-x86_64-msvc:
320-
MSYS_BITS: 64
321300
RUST_CONFIGURE_ARGS: >-
322301
--build=x86_64-pc-windows-msvc
323302
--target=x86_64-pc-windows-msvc,aarch64-pc-windows-msvc
@@ -327,7 +306,6 @@ jobs:
327306
DIST_REQUIRE_ALL_TOOLS: 1
328307
DEPLOY: 1
329308
dist-i686-msvc:
330-
MSYS_BITS: 32
331309
RUST_CONFIGURE_ARGS: >-
332310
--build=i686-pc-windows-msvc
333311
--target=i586-pc-windows-msvc
@@ -337,27 +315,20 @@ jobs:
337315
DIST_REQUIRE_ALL_TOOLS: 1
338316
DEPLOY: 1
339317
dist-i686-mingw:
340-
MSYS_BITS: 32
341318
RUST_CONFIGURE_ARGS: --build=i686-pc-windows-gnu --enable-full-tools --enable-profiler
342319
SCRIPT: python x.py dist
343-
MINGW_URL: https://rust-lang-ci-mirrors.s3-us-west-1.amazonaws.com/rustc
344-
MINGW_ARCHIVE: i686-6.3.0-release-posix-dwarf-rt_v5-rev2.7z
345-
MINGW_DIR: mingw32
320+
CUSTOM_MINGW: 1
346321
DIST_REQUIRE_ALL_TOOLS: 1
347322
DEPLOY: 1
348323
dist-x86_64-mingw:
349-
MSYS_BITS: 64
350324
SCRIPT: python x.py dist
351325
RUST_CONFIGURE_ARGS: --build=x86_64-pc-windows-gnu --enable-full-tools --enable-profiler
352-
MINGW_URL: https://rust-lang-ci-mirrors.s3-us-west-1.amazonaws.com/rustc
353-
MINGW_ARCHIVE: x86_64-6.3.0-release-posix-seh-rt_v5-rev2.7z
354-
MINGW_DIR: mingw64
326+
CUSTOM_MINGW: 1
355327
DIST_REQUIRE_ALL_TOOLS: 1
356328
DEPLOY: 1
357329

358330
# "alternate" deployment, see .travis.yml for more info
359331
dist-x86_64-msvc-alt:
360-
MSYS_BITS: 64
361332
RUST_CONFIGURE_ARGS: --build=x86_64-pc-windows-msvc --enable-extended --enable-profiler
362333
SCRIPT: python x.py dist
363334
DEPLOY_ALT: 1

Diff for: ‎src/ci/azure-pipelines/steps/run.yml

+10-45
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,13 @@
88

99
steps:
1010

11+
# Configure our CI_JOB_NAME variable which log analyzers can use for the main
12+
# step to see what's going on.
13+
- bash: |
14+
builder=$(echo $AGENT_JOBNAME | cut -d ' ' -f 2)
15+
echo "##vso[task.setvariable variable=CI_JOB_NAME]$builder"
16+
displayName: Configure Job Name
17+
1118
# Disable automatic line ending conversion, which is enabled by default on
1219
# Azure's Windows image. Having the conversion enabled caused regressions both
1320
# in our test suite (it broke miri tests) and in the ecosystem, since we
@@ -24,11 +31,8 @@ steps:
2431
- bash: src/ci/scripts/should-skip-this.sh
2532
displayName: Decide whether to run this job
2633

27-
# Spawn a background process to collect CPU usage statistics which we'll upload
28-
# at the end of the build. See the comments in the script here for more
29-
# information.
30-
- bash: python src/ci/cpu-usage-over-time.py &> cpu-usage.csv &
31-
displayName: "Collect CPU-usage statistics in the background"
34+
- bash: src/ci/scripts/collect-cpu-stats.sh
35+
displayName: Collect CPU-usage statistics in the background
3236

3337
- bash: src/ci/scripts/dump-environment.sh
3438
displayName: Show the current environment
@@ -135,48 +139,9 @@ steps:
135139
condition: and(succeeded(), not(variables.SKIP_JOB))
136140
displayName: Install awscli
137141

138-
# Configure our CI_JOB_NAME variable which log analyzers can use for the main
139-
# step to see what's going on.
140-
- bash: |
141-
builder=$(echo $AGENT_JOBNAME | cut -d ' ' -f 2)
142-
echo "##vso[task.setvariable variable=CI_JOB_NAME]$builder"
143-
displayName: Configure Job Name
144-
145-
# As a quick smoke check on the otherwise very fast mingw-check linux builder
146-
# check our own internal scripts.
147-
- bash: |
148-
set -e
149-
git clone --depth=1 https://github.com/rust-lang-nursery/rust-toolstate.git
150-
cd rust-toolstate
151-
python2.7 "$BUILD_SOURCESDIRECTORY/src/tools/publish_toolstate.py" "$(git rev-parse HEAD)" "$(git log --format=%s -n1 HEAD)" "" ""
152-
# Only check maintainers if this build is supposed to publish toolstate.
153-
# Builds that are not supposed to publish don't have the access token.
154-
if [ -n "${TOOLSTATE_PUBLISH+is_set}" ]; then
155-
TOOLSTATE_VALIDATE_MAINTAINERS_REPO=rust-lang/rust python2.7 "${BUILD_SOURCESDIRECTORY}/src/tools/publish_toolstate.py"
156-
fi
157-
cd ..
158-
rm -rf rust-toolstate
159-
env:
160-
TOOLSTATE_REPO_ACCESS_TOKEN: $(TOOLSTATE_REPO_ACCESS_TOKEN)
161-
condition: and(succeeded(), not(variables.SKIP_JOB), eq(variables['IMAGE'], 'mingw-check'))
162-
displayName: Verify the publish_toolstate script works
163-
164-
- bash: |
165-
set -e
166-
# Remove any preexisting rustup installation since it can interfere
167-
# with the cargotest step and its auto-detection of things like Clippy in
168-
# the environment
169-
rustup self uninstall -y || true
170-
if [ "$IMAGE" = "" ]; then
171-
src/ci/run.sh
172-
else
173-
src/ci/docker/run.sh $IMAGE
174-
fi
175-
#timeoutInMinutes: 180
142+
- bash: src/ci/scripts/run-build-from-ci.sh
176143
timeoutInMinutes: 600
177144
env:
178-
CI: true
179-
SRC: .
180145
AWS_ACCESS_KEY_ID: $(SCCACHE_AWS_ACCESS_KEY_ID)
181146
AWS_SECRET_ACCESS_KEY: $(SCCACHE_AWS_SECRET_ACCESS_KEY)
182147
TOOLSTATE_REPO_ACCESS_TOKEN: $(TOOLSTATE_REPO_ACCESS_TOKEN)

Diff for: ‎src/ci/azure-pipelines/try.yml

-1
Original file line numberDiff line numberDiff line change
@@ -72,7 +72,6 @@ jobs:
7272
# DEPLOY: 1
7373
#
7474
# dist-x86_64-msvc-alt:
75-
# MSYS_BITS: 64
7675
# RUST_CONFIGURE_ARGS: --build=x86_64-pc-windows-msvc --enable-extended --enable-profiler
7776
# SCRIPT: python x.py dist
7877
# DEPLOY_ALT: 1

Diff for: ‎src/ci/docker/armhf-gnu/Dockerfile

+1-1
Original file line numberDiff line numberDiff line change
@@ -72,7 +72,7 @@ RUN arm-linux-gnueabihf-gcc addentropy.c -o rootfs/addentropy -static
7272

7373
# TODO: What is this?!
7474
# Source of the file: https://github.com/vfdev-5/qemu-rpi2-vexpress/raw/master/vexpress-v2p-ca15-tc1.dtb
75-
RUN curl -O https://rust-lang-ci-mirrors.s3-us-west-1.amazonaws.com/rustc/vexpress-v2p-ca15-tc1.dtb
75+
RUN curl -O https://ci-mirrors.rust-lang.org/rustc/vexpress-v2p-ca15-tc1.dtb
7676

7777
COPY scripts/sccache.sh /scripts/
7878
RUN sh /scripts/sccache.sh

Diff for: ‎src/ci/docker/dist-armv7-linux/crosstool-ng.sh

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
set -ex
22

33
# Mirrored from https://github.com/crosstool-ng/crosstool-ng/archive/crosstool-ng-1.24.0.tar.gz
4-
url="https://rust-lang-ci-mirrors.s3-us-west-1.amazonaws.com/rustc/crosstool-ng-1.24.0.tar.gz"
4+
url="https://ci-mirrors.rust-lang.org/rustc/crosstool-ng-1.24.0.tar.gz"
55
curl -Lf $url | tar xzf -
66
cd crosstool-ng-crosstool-ng-1.24.0
77
./bootstrap

Diff for: ‎src/ci/docker/dist-various-1/install-mips-musl.sh

+1-1
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ mkdir /usr/local/mips-linux-musl
55
# originally from
66
# https://downloads.openwrt.org/snapshots/trunk/ar71xx/generic/
77
# OpenWrt-Toolchain-ar71xx-generic_gcc-5.3.0_musl-1.1.16.Linux-x86_64.tar.bz2
8-
URL="https://rust-lang-ci-mirrors.s3-us-west-1.amazonaws.com/rustc"
8+
URL="https://ci-mirrors.rust-lang.org/rustc"
99
FILE="OpenWrt-Toolchain-ar71xx-generic_gcc-5.3.0_musl-1.1.16.Linux-x86_64.tar.bz2"
1010
curl -L "$URL/$FILE" | tar xjf - -C /usr/local/mips-linux-musl --strip-components=2
1111

Diff for: ‎src/ci/docker/dist-various-1/install-mipsel-musl.sh

+1-1
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ mkdir /usr/local/mipsel-linux-musl
55
# Note that this originally came from:
66
# https://downloads.openwrt.org/snapshots/trunk/malta/generic/
77
# OpenWrt-Toolchain-malta-le_gcc-5.3.0_musl-1.1.15.Linux-x86_64.tar.bz2
8-
URL="https://rust-lang-ci-mirrors.s3-us-west-1.amazonaws.com/rustc"
8+
URL="https://ci-mirrors.rust-lang.org/rustc"
99
FILE="OpenWrt-Toolchain-malta-le_gcc-5.3.0_musl-1.1.15.Linux-x86_64.tar.bz2"
1010
curl -L "$URL/$FILE" | tar xjf - -C /usr/local/mipsel-linux-musl --strip-components=2
1111

Diff for: ‎src/ci/docker/dist-various-2/build-wasi-toolchain.sh

+1-1
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@
55
set -ex
66

77
# Originally from https://releases.llvm.org/9.0.0/clang+llvm-9.0.0-x86_64-linux-gnu-ubuntu-14.04.tar.xz
8-
curl https://rust-lang-ci-mirrors.s3-us-west-1.amazonaws.com/rustc/clang%2Bllvm-9.0.0-x86_64-linux-gnu-ubuntu-14.04.tar.xz | \
8+
curl https://ci-mirrors.rust-lang.org/rustc/clang%2Bllvm-9.0.0-x86_64-linux-gnu-ubuntu-14.04.tar.xz | \
99
tar xJf -
1010
export PATH=`pwd`/clang+llvm-9.0.0-x86_64-linux-gnu-ubuntu-14.04/bin:$PATH
1111

Diff for: ‎src/ci/docker/dist-x86_64-linux/build-curl.sh

+3
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,9 @@ source shared.sh
55

66
VERSION=7.66.0
77

8+
# This needs to be downloaded directly from S3, it can't go through the CDN.
9+
# That's because the CDN is backed by CloudFront, which requires SNI and TLSv1
10+
# (without paying an absurd amount of money).
811
curl https://rust-lang-ci-mirrors.s3-us-west-1.amazonaws.com/rustc/curl-$VERSION.tar.xz \
912
| xz --decompress \
1013
| tar xf -

Diff for: ‎src/ci/docker/dist-x86_64-linux/build-openssl.sh

+4
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,10 @@ set -ex
44
source shared.sh
55

66
VERSION=1.0.2k
7+
8+
# This needs to be downloaded directly from S3, it can't go through the CDN.
9+
# That's because the CDN is backed by CloudFront, which requires SNI and TLSv1
10+
# (without paying an absurd amount of money).
711
URL=https://rust-lang-ci-mirrors.s3-us-west-1.amazonaws.com/rustc/openssl-$VERSION.tar.gz
812

913
curl $URL | tar xzf -

Diff for: ‎src/ci/docker/dist-x86_64-netbsd/build-netbsd-toolchain.sh

+1-1
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@ cd netbsd
2525

2626
mkdir -p /x-tools/x86_64-unknown-netbsd/sysroot
2727

28-
URL=https://rust-lang-ci-mirrors.s3-us-west-1.amazonaws.com/rustc
28+
URL=https://ci-mirrors.rust-lang.org/rustc
2929

3030
# Originally from ftp://ftp.netbsd.org/pub/NetBSD/NetBSD-$BSD/source/sets/*.tgz
3131
curl $URL/2018-03-01-netbsd-src.tgz | tar xzf -

Diff for: ‎src/ci/docker/mingw-check/Dockerfile

+4-1
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,10 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
1919
COPY scripts/sccache.sh /scripts/
2020
RUN sh /scripts/sccache.sh
2121

22+
COPY mingw-check/validate-toolstate.sh /scripts/
23+
2224
ENV RUN_CHECK_WITH_PARALLEL_QUERIES 1
2325
ENV SCRIPT python2.7 ../x.py check --target=i686-pc-windows-gnu --host=i686-pc-windows-gnu && \
2426
python2.7 ../x.py build --stage 0 src/tools/build-manifest && \
25-
python2.7 ../x.py test --stage 0 src/tools/compiletest
27+
python2.7 ../x.py test --stage 0 src/tools/compiletest && \
28+
/scripts/validate-toolstate.sh

Diff for: ‎src/ci/docker/mingw-check/validate-toolstate.sh

+19
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
#!/bin/bash
2+
# A quick smoke test to make sure publish_tooolstate.py works.
3+
4+
set -euo pipefail
5+
IFS=$'\n\t'
6+
7+
rm -rf rust-toolstate
8+
git clone --depth=1 https://github.com/rust-lang-nursery/rust-toolstate.git
9+
cd rust-toolstate
10+
python2.7 "../../src/tools/publish_toolstate.py" "$(git rev-parse HEAD)" \
11+
"$(git log --format=%s -n1 HEAD)" "" ""
12+
# Only check maintainers if this build is supposed to publish toolstate.
13+
# Builds that are not supposed to publish don't have the access token.
14+
if [ -n "${TOOLSTATE_PUBLISH+is_set}" ]; then
15+
TOOLSTATE_VALIDATE_MAINTAINERS_REPO=rust-lang/rust python2.7 \
16+
"../../src/tools/publish_toolstate.py"
17+
fi
18+
cd ..
19+
rm -rf rust-toolstate

Diff for: ‎src/ci/docker/scripts/freebsd-toolchain.sh

+1-1
Original file line numberDiff line numberDiff line change
@@ -59,7 +59,7 @@ done
5959

6060
# Originally downloaded from:
6161
# https://download.freebsd.org/ftp/releases/${freebsd_arch}/${freebsd_version}-RELEASE/base.txz
62-
URL=https://rust-lang-ci-mirrors.s3-us-west-1.amazonaws.com/rustc/2019-04-04-freebsd-${freebsd_arch}-${freebsd_version}-RELEASE-base.txz
62+
URL=https://ci-mirrors.rust-lang.org/rustc/2019-04-04-freebsd-${freebsd_arch}-${freebsd_version}-RELEASE-base.txz
6363
curl "$URL" | tar xJf - -C "$sysroot" --wildcards "${files_to_extract[@]}"
6464

6565
# Fix up absolute symlinks from the system image. This can be removed

Diff for: ‎src/ci/docker/scripts/sccache.sh

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
set -ex
22

33
curl -fo /usr/local/bin/sccache \
4-
https://rust-lang-ci-mirrors.s3-us-west-1.amazonaws.com/rustc/2018-04-02-sccache-x86_64-unknown-linux-musl
4+
https://ci-mirrors.rust-lang.org/rustc/2018-04-02-sccache-x86_64-unknown-linux-musl
55

66
chmod +x /usr/local/bin/sccache

Diff for: ‎src/ci/scripts/collect-cpu-stats.sh

+9
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
#!/bin/bash
2+
# Spawn a background process to collect CPU usage statistics which we'll upload
3+
# at the end of the build. See the comments in the script here for more
4+
# information.
5+
6+
set -euo pipefail
7+
IFS=$'\n\t'
8+
9+
python src/ci/cpu-usage-over-time.py &> cpu-usage.csv &

Diff for: ‎src/ci/scripts/install-clang.sh

+1-1
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ if isMacOS; then
1717
# Configure `AR` specifically so rustbuild doesn't try to infer it as
1818
# `clang-ar` by accident.
1919
ciCommandSetEnv AR "ar"
20-
elif isWindows && [[ -z ${MINGW_URL+x} ]]; then
20+
elif isWindows && [[ ${CUSTOM_MINGW-0} -ne 1 ]]; then
2121
# If we're compiling for MSVC then we, like most other distribution builders,
2222
# switch to clang as the compiler. This'll allow us eventually to enable LTO
2323
# amongst LLVM and rustc. Note that we only do this on MSVC as I don't think

0 commit comments

Comments
 (0)