Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Debugging information corrupt #66191

Closed
MichalStrehovsky opened this issue Mar 4, 2022 · 2 comments · Fixed by #75590
Closed

Debugging information corrupt #66191

MichalStrehovsky opened this issue Mar 4, 2022 · 2 comments · Fixed by #75590
Labels
area-NativeAOT-coreclr
Milestone
Future

Comments

@MichalStrehovsky
Copy link
Member

#66061 (comment)

Looks related to multimodule mode that is not shipping.

MultiModule.obj : warning LNK4078: multiple '.bss' sections found with different attributes (C0401040) [D:\a\_work\1\s\src\tests\nativeaot\SmokeTests\MultiModule\MultiModule.csproj] [D:\a\_work\1\s\src\tests\build.proj]
Framework.lib(System.Private.Reflection.Core.obj) : warning LNK4209: debugging information corrupt; recompile module; linking object as if no debug info [D:\a\_work\1\s\src\tests\nativeaot\SmokeTests\MultiModule\MultiModule.csproj] [D:\a\_work\1\s\src\tests\build.proj]
Framework.lib(System.Private.Reflection.Metadata.obj) : warning LNK4209: debugging information corrupt; recompile module; linking object as if no debug info [D:\a\_work\1\s\src\tests\nativeaot\SmokeTests\MultiModule\MultiModule.csproj] [D:\a\_work\1\s\src\tests\build.proj]
Framework.lib(System.Private.TypeLoader.obj) : warning LNK4209: debugging information corrupt; recompile module; linking object as if no debug info [D:\a\_work\1\s\src\tests\nativeaot\SmokeTests\MultiModule\MultiModule.csproj] [D:\a\_work\1\s\src\tests\build.proj]
Framework.lib(System.Private.Interop.obj) : warning LNK4209: debugging information corrupt; recompile module; linking object as if no debug info [D:\a\_work\1\s\src\tests\nativeaot\SmokeTests\MultiModule\MultiModule.csproj] [D:\a\_work\1\s\src\tests\build.proj]
Framework.lib(System.Private.Reflection.Execution.obj) : warning LNK4209: debugging information corrupt; recompile module; linking object as if no debug info [D:\a\_work\1\s\src\tests\nativeaot\SmokeTests\MultiModule\MultiModule.csproj] [D:\a\_work\1\s\src\tests\build.proj]
Framework.lib(System.Private.StackTraceMetadata.obj) : warning LNK4209: debugging information corrupt; recompile module; linking object as if no debug info [D:\a\_work\1\s\src\tests\nativeaot\SmokeTests\MultiModule\MultiModule.csproj] [D:\a\_work\1\s\src\tests\build.proj]
LINK : fatal error LNK1318: Unexpected PDB error; OK (0) '' [D:\a\_work\1\s\src\tests\nativeaot\SmokeTests\MultiModule\MultiModule.csproj] [D:\a\_work\1\s\src\tests\build.proj]
@dotnet-issue-labeler dotnet-issue-labeler bot added the untriaged New issue has not been triaged by the area owner label Mar 4, 2022
@danmoseley danmoseley added the blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' label Mar 4, 2022
@runfoapp runfoapp bot mentioned this issue Mar 4, 2022
Closed
MichalStrehovsky added a commit that referenced this issue Mar 7, 2022
@MichalStrehovsky
Add MultiModule test to issues.targets
629a629 
Hitting #66191 in the CI.
@MichalStrehovsky MichalStrehovsky mentioned this issue Mar 7, 2022
Merged
@BruceForstall
Copy link
Member

fyi, one example:

https://dev.azure.com/dnceng/public/_build/results?buildId=1649847&view=logs&j=6db13b2b-91c6-5dc1-932a-ee0a68560d9e&t=4b34605a-5eef-5b38-e2cc-6f633df186e0

MichalStrehovsky added a commit that referenced this issue Mar 8, 2022
@MichalStrehovsky
Add MultiModule test to issues.targets (#66289)
34880aa 
Hitting #66191 in the CI.
@kunalspathak kunalspathak mentioned this issue Mar 16, 2022
Merged
MichalStrehovsky added a commit to MichalStrehovsky/runtime that referenced this issue Mar 24, 2022
@MichalStrehovsky
Stabilize summary blobs
5d51a4e 
I was trying to repro dotnet#66191 and assumed determinism is the root cause. I used `CoreRT_DeterminismSeed` to perturb the dependency node evaluation and it found differences.

Now, I don't think the differences really matter. We already seem to be deterministic in the multithreaded compilation (the outputs are always the same no matter how many times I try).

`CoreRT_DeterminismSeed` perturbs the order in which node dependencies are evaluated and that exposes more ordering issues than we practically have.

In normal multithreaded compilation we grab nodes in different order from the `NodeFactory` as part of method compilation, but the order in which those nodes are evaluated is still fixed. This means the nodes we grabbed are still getting marked in a deterministic order and we get stable ordering even without an explicit sort in this spot.

But maybe as part of https://github.com/dotnet/runtimelab/issues/1631 we should enable `CoreRT_Determinism` in our testing and for that we need this extra sorting.
@MichalStrehovsky MichalStrehovsky mentioned this issue Mar 24, 2022
Merged
jkotas pushed a commit that referenced this issue Mar 25, 2022
@MichalStrehovsky
Stabilize summary blobs (#67086)
343d8ee 
I was trying to repro #66191 and assumed determinism is the root cause. I used `CoreRT_DeterminismSeed` to perturb the dependency node evaluation and it found differences.

Now, I don't think the differences really matter. We already seem to be deterministic in the multithreaded compilation (the outputs are always the same no matter how many times I try).

`CoreRT_DeterminismSeed` perturbs the order in which node dependencies are evaluated and that exposes more ordering issues than we practically have.

In normal multithreaded compilation we grab nodes in different order from the `NodeFactory` as part of method compilation, but the order in which those nodes are evaluated is still fixed. This means the nodes we grabbed are still getting marked in a deterministic order and we get stable ordering even without an explicit sort in this spot.

But maybe as part of https://github.com/dotnet/runtimelab/issues/1631 we should enable `CoreRT_Determinism` in our testing and for that we need this extra sorting.
@MichalStrehovsky MichalStrehovsky removed blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' untriaged New issue has not been triaged by the area owner labels Mar 30, 2022
@MichalStrehovsky MichalStrehovsky added this to the Future milestone Mar 30, 2022
radekdoulik pushed a commit to radekdoulik/runtime that referenced this issue Mar 30, 2022
@MichalStrehovsky @radekdoulik
Stabilize summary blobs (dotnet#67086)
956bcdc 
I was trying to repro dotnet#66191 and assumed determinism is the root cause. I used `CoreRT_DeterminismSeed` to perturb the dependency node evaluation and it found differences.

Now, I don't think the differences really matter. We already seem to be deterministic in the multithreaded compilation (the outputs are always the same no matter how many times I try).

`CoreRT_DeterminismSeed` perturbs the order in which node dependencies are evaluated and that exposes more ordering issues than we practically have.

In normal multithreaded compilation we grab nodes in different order from the `NodeFactory` as part of method compilation, but the order in which those nodes are evaluated is still fixed. This means the nodes we grabbed are still getting marked in a deterministic order and we get stable ordering even without an explicit sort in this spot.

But maybe as part of https://github.com/dotnet/runtimelab/issues/1631 we should enable `CoreRT_Determinism` in our testing and for that we need this extra sorting.
@MichalStrehovsky
Copy link
Member Author

We have an instance of this in nativeaot-llvm branch that hasn't seen a non-WASM change since ~September. https://dev.azure.com/dnceng/public/_build/results?buildId=1700534&view=logs&j=882c4811-6d19-5cba-ad52-4ad8914abd7d&t=5042405d-70a2-53fe-87f6-fb9d1d6c8de0. The failure is definitely new.

This very strongly suggest this is caused by something in the environment. Link.exe version or something like that.

@MichalStrehovsky MichalStrehovsky mentioned this issue Jul 20, 2022
Merged
MichalStrehovsky added a commit to MichalStrehovsky/runtime that referenced this issue Sep 14, 2022
@MichalStrehovsky
Re-enable multimodule testing
6b1b918 
Resolves dotnet#66191.

I no longer see the warning or the link.exe failure. This likely was a link.exe bug like I suspected in the issue based on available evidence.

Of course we also regressed multimodule mode so I'm fixing that too.
@MichalStrehovsky MichalStrehovsky mentioned this issue Sep 14, 2022
Merged
@ghost ghost added the in-pr There is an active PR which will close this issue when it is merged label Sep 14, 2022
jkotas pushed a commit that referenced this issue Sep 14, 2022
@MichalStrehovsky
Re-enable multimodule testing (#75590)
0ff1f89 
Resolves #66191.

I no longer see the warning or the link.exe failure. This likely was a link.exe bug like I suspected in the issue based on available evidence.

Of course we also regressed multimodule mode so I'm fixing that too.
@ghost ghost removed the in-pr There is an active PR which will close this issue when it is merged label Sep 14, 2022
@ghost ghost locked as resolved and limited conversation to collaborators Oct 14, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
area-NativeAOT-coreclr
Projects
None yet
Milestone
Future
Development

Successfully merging a pull request may close this issue.

Re-enable multimodule testing MichalStrehovsky/runtime
3 participants
@danmoseley @BruceForstall @MichalStrehovsky