Workaround for windows-gnu rust-lld test failure #140396

ChrisDenton · 2025-04-28T14:21:56Z

The test run-make/amdgpu-kd has an issue on windows-gnu where rust-lld will sometimes fail with error 0xc0000374 (STATUS_HEAP_CORRUPTION).

This works around the issue by passing --threads=1 to the linker as suggested here. Note I don't know if this will help and it happens only sometimes in our CI so it's hard to test.

rustbot · 2025-04-28T14:22:02Z

r? @jieyouxu

rustbot has assigned @jieyouxu.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

rustbot · 2025-04-28T14:22:05Z

This PR modifies run-make tests.

cc @jieyouxu

ChrisDenton · 2025-04-28T14:22:15Z

@bors try

Workaround for windows-gnu rust-lld test failure The test run-make/amdgpu-kd has an issue on windows-gnu where rust-lld will sometimes fail with error 0xc0000374 (`STATUS_HEAP_CORRUPTION`). This works around the issue by passing `--threads=1` to the linker as suggested [here](rust-lang#115985 (comment)). Note I don't know if this will help and it happens only sometimes in our CI so it's hard to test. try-job: x86_64-mingw-1

bors · 2025-04-28T14:23:27Z

⌛ Trying commit b107e98 with merge 46b91f6...

jieyouxu

Yeah, this is one of the tests where this can happen, but there's also other tests that run into rust-lld heap corruption.

r=me if the try-job comes back green, I guess even r=me if it fails due to heap corruption anyway (hopefully single thread makes it less likely)...

tests/run-make/amdgpu-kd/rmake.rs

petrochenkov · 2025-04-28T14:28:23Z

@ChrisDenton Have you seen the RUSTC_RETRY_LINKER_ON_SEGFAULT hack in the compiler? 😄
If the issue may appear on something different than tests/run-make/amdgpu-kd/rmake.rs then something like that may make sense on Windows as well.

jieyouxu · 2025-04-28T14:29:28Z

Lmao I did not know that was a thing

ChrisDenton · 2025-04-28T14:31:29Z

Me neither, ha.

Though this issue does seem to be specific to this windows-gnu, rust-lld and this test in particular. I looked at some of the older test fails and they don't seem to exist any more (I guess this test replaced them?). Like tests/ui/amdgpu-require-explicit-cpu.rs

jieyouxu · 2025-04-28T14:34:23Z

I believe there's also avr-rjmp-offset where we disabled the test on windows-gnu because of rust-lld too (#133480)

ChrisDenton · 2025-04-28T14:37:46Z

Ah, that's already disabled so I didn't see it in the logs. Also it is a STATUS_ACCESS_VIOLATION instead of a STATUS_HEAP_CORRUPTION but it possibly has a similar cause.

jieyouxu · 2025-04-28T14:38:03Z

@ChrisDenton Have you seen the RUSTC_RETRY_LINKER_ON_SEGFAULT hack in the compiler? 😄 If the issue may appear on something different than tests/run-make/amdgpu-kd/rmake.rs then something like that may make sense on Windows as well.

Apparently that was for a similarly cursed problem, #38878.

rust/compiler/rustc_codegen_ssa/src/back/link.rs

Lines 842 to 861 in a932eb3

    
           // Here's a terribly awful hack that really shouldn't be present in any 
        
           // compiler. Here an environment variable is supported to automatically 
        
           // retry the linker invocation if the linker looks like it segfaulted. 
        
           // 
        
           // Gee that seems odd, normally segfaults are things we want to know 
        
           // about!  Unfortunately though in rust-lang/rust#38878 we're 
        
           // experiencing the linker segfaulting on Travis quite a bit which is 
        
           // causing quite a bit of pain to land PRs when they spuriously fail 
        
           // due to a segfault. 
        
           // 
        
           // The issue #38878 has some more debugging information on it as well, 
        
           // but this unfortunately looks like it's just a race condition in 
        
           // macOS's linker with some thread pool working in the background. It 
        
           // seems that no one currently knows a fix for this so in the meantime 
        
           // we're left with this... 
        
           if !retry_on_segfault || i > 3 { 
        
               break; 
        
           } 
        
           let msg_segv = "clang: error: unable to execute command: Segmentation fault: 11"; 
        
           let msg_bus = "clang: error: unable to execute command: Bus error: 10";

ChrisDenton · 2025-04-28T14:59:47Z

So I think I'd personally prefer not to pile on the hack if it can be avoided. If this PR fixes the issue and we don't get more cropping up then I'd like to leave that alone. If it does crop up again then adding 0xc0000374 and 0xc0000005 to the hack would be the least worst option, I guess.

I do wonder if we could work around this in run-make. The thing the avr test and this one have in common is cross-compiling for a more specialised target. But I'd like to first confirm that setting --threads=1 does in fact help.

jieyouxu · 2025-04-28T15:40:29Z

Yeah, that is reasonable

jieyouxu · 2025-04-28T15:47:03Z

@bors r+ rollup

bors · 2025-04-28T15:47:06Z

📌 Commit 52594ef has been approved by jieyouxu

It is now in the queue for this repository.

ChrisDenton · 2025-04-28T16:14:33Z

That seems to an entirely different spurious failure 😅. A stack overflow in mir-opt\building\issue_49232.rs, We've had other such spurious errors before but they seem to come and go, e.g. #138110

jieyouxu · 2025-04-28T16:31:48Z

That is... yeah.

jieyouxu · 2025-04-28T16:32:39Z

Oh I didn't realize the try-job didn't finish... Just in case
@bors r-

jieyouxu · 2025-04-28T16:32:48Z

@bors r+

bors · 2025-04-28T16:32:51Z

📌 Commit 52594ef has been approved by jieyouxu

It is now in the queue for this repository.

Workaround for windows-gnu rust-lld test failure The test run-make/amdgpu-kd has an issue on windows-gnu where rust-lld will sometimes fail with error 0xc0000374 (`STATUS_HEAP_CORRUPTION`). This works around the issue by passing `--threads=1` to the linker as suggested [here](rust-lang#115985 (comment)). Note I don't know if this will help and it happens only sometimes in our CI so it's hard to test.

…enton Rollup of 9 pull requests Successful merges: - rust-lang#139308 (add autodiff inline) - rust-lang#140276 (Do not compute type_of for impl item if impl where clauses are unsatisfied) - rust-lang#140302 (Move inline asm check to typeck, properly handle aliases) - rust-lang#140323 (Implement the internal feature `cfg_target_has_reliable_f16_f128`) - rust-lang#140374 (Resolve instance for SymFn in global/naked asm) - rust-lang#140391 (Rename sub_ptr to offset_from_unsigned in docs) - rust-lang#140394 (Make bootstrap git tests more self-contained) - rust-lang#140396 (Workaround for windows-gnu rust-lld test failure) - rust-lang#140402 (only return nested goals for `Certainty::Yes`) r? `@ghost` `@rustbot` modify labels: rollup

Workaround for windows-gnu rust-lld test failure The test run-make/amdgpu-kd has an issue on windows-gnu where rust-lld will sometimes fail with error 0xc0000374 (`STATUS_HEAP_CORRUPTION`). This works around the issue by passing `--threads=1` to the linker as suggested [here](rust-lang#115985 (comment)). Note I don't know if this will help and it happens only sometimes in our CI so it's hard to test.

…enton Rollup of 10 pull requests Successful merges: - rust-lang#139308 (add autodiff inline) - rust-lang#139656 (Stabilize `slice_as_chunks` library feature) - rust-lang#140022 (allow deref patterns to move out of boxes) - rust-lang#140276 (Do not compute type_of for impl item if impl where clauses are unsatisfied) - rust-lang#140302 (Move inline asm check to typeck, properly handle aliases) - rust-lang#140323 (Implement the internal feature `cfg_target_has_reliable_f16_f128`) - rust-lang#140391 (Rename sub_ptr to offset_from_unsigned in docs) - rust-lang#140394 (Make bootstrap git tests more self-contained) - rust-lang#140396 (Workaround for windows-gnu rust-lld test failure) - rust-lang#140402 (only return nested goals for `Certainty::Yes`) Failed merges: - rust-lang#139765 ([beta] Delay `hash_extract_if` stabilization from 1.87 to 1.88) r? `@ghost` `@rustbot` modify labels: rollup

The test run-make/amdgpu-kd has an issue where rust-lld will sometimes fail with error 0xc0000374 (STATUS_HEAP_CORRUPTION).

ChrisDenton · 2025-04-28T23:27:48Z

Oh, oops, I need to be more explicit about types. I forgot it's ignored on my local machine

@bors r=jieyouxu

bors · 2025-04-28T23:27:50Z

📌 Commit 3c42dc2 has been approved by jieyouxu

It is now in the queue for this repository.

…enton Rollup of 10 pull requests Successful merges: - rust-lang#139308 (add autodiff inline) - rust-lang#139656 (Stabilize `slice_as_chunks` library feature) - rust-lang#140022 (allow deref patterns to move out of boxes) - rust-lang#140276 (Do not compute type_of for impl item if impl where clauses are unsatisfied) - rust-lang#140302 (Move inline asm check to typeck, properly handle aliases) - rust-lang#140323 (Implement the internal feature `cfg_target_has_reliable_f16_f128`) - rust-lang#140391 (Rename sub_ptr to offset_from_unsigned in docs) - rust-lang#140394 (Make bootstrap git tests more self-contained) - rust-lang#140396 (Workaround for windows-gnu rust-lld test failure) - rust-lang#140402 (only return nested goals for `Certainty::Yes`) r? `@ghost` `@rustbot` modify labels: rollup

Rollup merge of rust-lang#140396 - ChrisDenton:gnu-threads, r=jieyouxu Workaround for windows-gnu rust-lld test failure The test run-make/amdgpu-kd has an issue on windows-gnu where rust-lld will sometimes fail with error 0xc0000374 (`STATUS_HEAP_CORRUPTION`). This works around the issue by passing `--threads=1` to the linker as suggested [here](rust-lang#115985 (comment)). Note I don't know if this will help and it happens only sometimes in our CI so it's hard to test.

rustbot assigned jieyouxu Apr 28, 2025

rustbot added A-run-make Area: port run-make Makefiles to rmake.rs S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Apr 28, 2025

jieyouxu approved these changes Apr 28, 2025

View reviewed changes

jieyouxu reviewed Apr 28, 2025

View reviewed changes

tests/run-make/amdgpu-kd/rmake.rs Outdated Show resolved Hide resolved

ChrisDenton force-pushed the gnu-threads branch from b107e98 to cb615c0 Compare April 28, 2025 14:28

This comment has been minimized.

Sign in to view

ChrisDenton force-pushed the gnu-threads branch from cb615c0 to 52594ef Compare April 28, 2025 14:41

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 28, 2025

This comment has been minimized.

Sign in to view

bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Apr 28, 2025

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Apr 28, 2025

ChrisDenton mentioned this pull request Apr 28, 2025

Rollup of 9 pull requests #140411

Closed

ChrisDenton mentioned this pull request Apr 28, 2025

Rollup of 10 pull requests #140414

Closed

Workaround for windows-gnu rust-lld test failure

3c42dc2

The test run-make/amdgpu-kd has an issue where rust-lld will sometimes fail with error 0xc0000374 (STATUS_HEAP_CORRUPTION).

ChrisDenton force-pushed the gnu-threads branch from 52594ef to 3c42dc2 Compare April 28, 2025 23:25

ChrisDenton mentioned this pull request Apr 28, 2025

Rollup of 10 pull requests #140415

Merged

bors merged commit 469f03d into rust-lang:master Apr 29, 2025
6 checks passed

rustbot added this to the 1.88.0 milestone Apr 29, 2025

ChrisDenton deleted the gnu-threads branch April 29, 2025 08:50

Workaround for windows-gnu rust-lld test failure #140396

Workaround for windows-gnu rust-lld test failure #140396

Uh oh!

Conversation

ChrisDenton commented Apr 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rustbot commented Apr 28, 2025

Uh oh!

rustbot commented Apr 28, 2025

Uh oh!

ChrisDenton commented Apr 28, 2025

Uh oh!

bors commented Apr 28, 2025

Uh oh!

jieyouxu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

petrochenkov commented Apr 28, 2025

Uh oh!

jieyouxu commented Apr 28, 2025

Uh oh!

ChrisDenton commented Apr 28, 2025

Uh oh!

jieyouxu commented Apr 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ChrisDenton commented Apr 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jieyouxu commented Apr 28, 2025

Uh oh!

This comment has been minimized.

ChrisDenton commented Apr 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jieyouxu commented Apr 28, 2025

Uh oh!

jieyouxu commented Apr 28, 2025

Uh oh!

bors commented Apr 28, 2025

Uh oh!

This comment has been minimized.

ChrisDenton commented Apr 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jieyouxu commented Apr 28, 2025

Uh oh!

jieyouxu commented Apr 28, 2025

Uh oh!

jieyouxu commented Apr 28, 2025

Uh oh!

bors commented Apr 28, 2025

Uh oh!

ChrisDenton commented Apr 28, 2025

Uh oh!

bors commented Apr 28, 2025

Uh oh!

Uh oh!

Uh oh!

ChrisDenton commented Apr 28, 2025 •

edited

Loading

jieyouxu commented Apr 28, 2025 •

edited

Loading

ChrisDenton commented Apr 28, 2025 •

edited

Loading

ChrisDenton commented Apr 28, 2025 •

edited

Loading

ChrisDenton commented Apr 28, 2025 •

edited

Loading