Builds often timeout due to no console output activity #83329
Description
Description
I have encountered numerous builds that filed due to the pipeline not receiving any console output. Re-triggering the is the workaround, but it's not ideal as pipeline builds take hours to run.
Additional information
Here are some builds that I've come across
- https://ci.swift.org/job/swift-PR-macos-smoke-test/21458/console (around July 9th, 2025)
09:56:43 + /usr/local/bin/sccache --stop-server 09:56:43 Stopping sccache server... 11:02:43 Cancelling nested steps due to timeout 11:02:43 Sending interrupt signal to process - https://ci.swift.org/job/swift-PR-macos-smoke-test/21461/console (around July 9th, 2025)
12:21:45 [1126/5293][ 21%][72.331s] /usr/local/bin/sccache /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++ -DGTEST_HAS_RTTI=0 -D_DEBUG -D_GLIBCXX_ASSERTIONS -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/Users/ec2-user/jenkins/workspace/swift-PR-macos-smoke-test/branch-main/build/buildbot_incremental/llvm-macosx-x86_64/lib/Transforms/Vectorize -I/Users/ec2-user/jenkins/workspace/swift-PR-macos-smoke-test/branch-main/llvm-project/llvm/lib/Transforms/Vectorize -I/Users/ec2-user/jenkins/workspace/swift-PR-macos-smoke-test/branch-main/build/buildbot_incremental/llvm-macosx-x86_64/include -I/Users/ec2-user/jenkins/workspace/swift-PR-macos-smoke-test/branch-main/llvm-project/llvm/include -Wno-unknown-warning-option -Werror=unguarded-availability-new -arch x86_64 -target x86_64-apple-macosx13.0 -fno-stack-protector -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -O3 -DNDEBUG -std=c++17 -arch x86_64 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX15.2.sdk -mmacosx-version-min=13.0 -fno-exceptions -funwind-tables -fno-rtti -UNDEBUG -MD -MT lib/Transforms/Vectorize/CMakeFiles/LLVMVectorize.dir/LoopVectorize.cpp.o -MF lib/Transforms/Vectorize/CMakeFiles/LLVMVectorize.dir/LoopVectorize.cpp.o.d -o lib/Transforms/Vectorize/CMakeFiles/LLVMVectorize.dir/LoopVectorize.cpp.o -c /Users/ec2-user/jenkins/workspace/swift-PR-macos-smoke-test/branch-main/llvm-project/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp 13:22:38 Cancelling nested steps due to timeout - https://ci.swift.org/job/swift-PR-macos-smoke-test/21465 (around July 9th, 2025)
14:13:17 [543/5293][ 10%][31.159s] /usr/local/bin/sccache /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++ -DGTEST_HAS_RTTI=0 -D_DEBUG -D_GLIBCXX_ASSERTIONS -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/Users/ec2-user/jenkins/workspace/swift-PR-macos-smoke-test/branch-main/build/buildbot_incremental/llvm-macosx-x86_64/lib/CodeGen -I/Users/ec2-user/jenkins/workspace/swift-PR-macos-smoke-test/branch-main/llvm-project/llvm/lib/CodeGen -I/Users/ec2-user/jenkins/workspace/swift-PR-macos-smoke-test/branch-main/build/buildbot_incremental/llvm-macosx-x86_64/include -I/Users/ec2-user/jenkins/workspace/swift-PR-macos-smoke-test/branch-main/llvm-project/llvm/include -Wno-unknown-warning-option -Werror=unguarded-availability-new -arch x86_64 -target x86_64-apple-macosx13.0 -fno-stack-protector -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -O3 -DNDEBUG -std=c++17 -arch x86_64 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX15.2.sdk -mmacosx-version-min=13.0 -fno-exceptions -funwind-tables -fno-rtti -UNDEBUG -MD -MT lib/CodeGen/CMakeFiles/LLVMCodeGen.dir/ExpandReductions.cpp.o -MF lib/CodeGen/CMakeFiles/LLVMCodeGen.dir/ExpandReductions.cpp.o.d -o lib/CodeGen/CMakeFiles/LLVMCodeGen.dir/ExpandReductions.cpp.o -c /Users/ec2-user/jenkins/workspace/swift-PR-macos-smoke-test/branch-main/llvm-project/llvm/lib/CodeGen/ExpandReductions.cpp 15:14:49 Cancelling nested steps due to timeout 15:14:49 Sending interrupt signal to process - https://ci.swift.org/job/swift-package-manager-PR-macos-smoke-test/6703 (around July 16th, 2025)
11:36:26 [5410/5465][ 98%][374.497s] : && "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool" -static -no_warning_for_no_symbols -o lib/libbenchmark_main.a third-party/benchmark/src/CMakeFiles/benchmark_main.dir/benchmark_main.cc.o && : 12:48:46 Cancelling nested steps due to timeout 12:48:46 Sending interrupt signal to process 12:48:50 ninja: build stopped: interrupted by user. - https://ci.swift.org/job/swift-package-manager-PR-macos-smoke-test/6742 (July 25, 2025)
11:02:39 [5256/5465][ 96%][379.938s] /usr/local/bin/sccache /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++ -DBENCHMARK_STATIC_DEFINE -DHAVE_STD_REGEX -D_DEBUG -D_FILE_OFFSET_BITS=64 -D_GLIBCXX_ASSERTIONS -D_LARGEFILE64_SOURCE -D_LARGEFILE_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/Users/ec2-user/jenkins/workspace/swift-package-manager-PR-macos-smoke-test/branch-main/build/buildbot_incremental/llvm-macosx-arm64/third-party/benchmark/src -I/Users/ec2-user/jenkins/workspace/swift-package-manager-PR-macos-smoke-test/branch-main/llvm-project/third-party/benchmark/src -I/Users/ec2-user/jenkins/workspace/swift-package-manager-PR-macos-smoke-test/branch-main/build/buildbot_incremental/llvm-macosx-arm64/include -I/Users/ec2-user/jenkins/workspace/swift-package-manager-PR-macos-smoke-test/branch-main/llvm-project/llvm/include -I/Users/ec2-user/jenkins/workspace/swift-package-manager-PR-macos-smoke-test/branch-main/llvm-project/third-party/benchmark/include -Wno-unknown-warning-option -Werror=unguarded-availability-new -arch arm64 -target arm64-apple-macosx13.0 -fno-stack-protector -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -Wall -Wextra -Wshadow -Wfloat-equal -Wold-style-cast -Wsuggest-override -pedantic -pedantic-errors -Wshorten-64-to-32 -fstrict-aliasing -Wno-deprecated-declarations -fno-exceptions -Wstrict-aliasing -Wthread-safety -O3 -DNDEBUG -std=c++11 -arch arm64 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX15.2.sdk -mmacosx-version-min=13.0 -fvisibility=hidden -fvisibility-inlines-hidden -UNDEBUG -MD -MT third-party/benchmark/src/CMakeFiles/benchmark.dir/timers.cc.o -MF third-party/benchmark/src/CMakeFiles/benchmark.dir/timers.cc.o.d -o third-party/benchmark/src/CMakeFiles/benchmark.dir/timers.cc.o -c /Users/ec2-user/jenkins/workspace/swift-package-manager-PR-macos-smoke-test/branch-main/llvm-project/third-party/benchmark/src/timers.cc 11:02:39 [5257/5465][ 96%][379.982s] : && "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool" -static -no_warning_for_no_symbols -o lib/libbenchmark.a third-party/benchmark/src/CMakeFiles/benchmark.dir/benchmark.cc.o third-party/benchmark/src/CMakeFiles/benchmark.dir/benchmark_api_internal.cc.o third-party/benchmark/src/CMakeFiles/benchmark.dir/benchmark_name.cc.o third-party/benchmark/src/CMakeFiles/benchmark.dir/benchmark_register.cc.o third-party/benchmark/src/CMakeFiles/benchmark.dir/benchmark_runner.cc.o third-party/benchmark/src/CMakeFiles/benchmark.dir/check.cc.o third-party/benchmark/src/CMakeFiles/benchmark.dir/colorprint.cc.o third-party/benchmark/src/CMakeFiles/benchmark.dir/commandlineflags.cc.o third-party/benchmark/src/CMakeFiles/benchmark.dir/complexity.cc.o third-party/benchmark/src/CMakeFiles/benchmark.dir/console_reporter.cc.o third-party/benchmark/src/CMakeFiles/benchmark.dir/counter.cc.o third-party/benchmark/src/CMakeFiles/benchmark.dir/csv_reporter.cc.o third-party/benchmark/src/CMakeFiles/benchmark.dir/json_reporter.cc.o third-party/benchmark/src/CMakeFiles/benchmark.dir/perf_counters.cc.o third-party/benchmark/src/CMakeFiles/benchmark.dir/reporter.cc.o third-party/benchmark/src/CMakeFiles/benchmark.dir/statistics.cc.o third-party/benchmark/src/CMakeFiles/benchmark.dir/string_util.cc.o third-party/benchmark/src/CMakeFiles/benchmark.dir/sysinfo.cc.o third-party/benchmark/src/CMakeFiles/benchmark.dir/timers.cc.o && : 11:02:39 [5258/5465][ 96%][380.000s] : && "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool" -static -no_warning_for_no_symbols -o lib/libbenchmark_main.a third-party/benchmark/src/CMakeFiles/benchmark_main.dir/benchmark_main.cc.o && : 12:14:41 Cancelling nested steps due to timeout 12:14:41 Sending interrupt signal to process 12:14:48 ninja: build stopped: interrupted by user. 12:14:48 /Users/ec2-user/jenkins/workspace/swift-package-manager-PR-macos-smoke-test@tmp/durable-b81ffe0d/script.sh.copy: line 1: 35243 Terminated: 15 /Users/ec2-user/jenkins/workspace/swift-package-manager-PR-macos-smoke-test/branch-main/swift/utils/build-script --preset=buildbot_swiftpm_macos_platform,tools=RA,stdlib=RA --cmake-c-launcher /usr/local/bin/sccache --cmake-cxx-launcher /usr/local/bin/sccache 12:14:48 script returned exit code 143