Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tracking issue for -Zrun-dsymutil #79361

Closed
alexcrichton opened this issue Nov 23, 2020 · 3 comments · Fixed by #79570
Closed

Tracking issue for -Zrun-dsymutil #79361

alexcrichton opened this issue Nov 23, 2020 · 3 comments · Fixed by #79570
Labels
A-debuginfo Area: Debugging information in compiled programs (DWARF, PDB, etc.) A-runtime Area: std's runtime and "pre-main" init for handling backtraces, unwinds, stack overflows C-tracking-issue Category: An issue tracking the progress of sth. like the implementation of an RFC I-compiletime Issue: Problems and improvements with respect to compile times. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.

Comments

@alexcrichton
Copy link
Member

This is a (belatedly-opened) issue with the intention of tracking the -Zrun-dsymutil flag to the compiler. This flag was implemented long ago in #47784 and the intention is to prevent rustc from automatically running dsymutil on macOS, which it automatically does today for builds with debuginfo. The purpose for doing this is that dsymutil is a pretty slow process and can make debug incremental compile times quite painful.

The purpose of dsymutil itself is to collect all debuginfo from the executable and all its dependencies into an adjacent *.dSYM folder next to the executable itself. This form of split-debuginfo is then read by debuggers and such. This consequently means that dsymutil is inherently not incremental. It will unconditionally run over the entire final binary and produce the entire *.dSYM folder each time. Since this runs over the final artifact which typically has a lot of debuginfo this can take quite a long time.

As discovered on #47784, however, the reason this has been unstable for so long is that dsymutil was required for filenames and line numbers with RUST_BACKTRACE by default. It was too unreasonable for Cargo or other tooling to not run dsymutil because it broke this important feature. With the advent of a Rust-based backtrace backend, however, we now have support to parse debuginfo even if dsymutil hasn't run. This uses the same technique LLDB uses to give you debuginfo even when dsymutil hasn't run. This was then later integrated into libstd and is purportedly working well and greatly improving incremental compile time.

As for the nitty-gritty of what this flag does:

  • If enabled run-dsymutil will execute dsymutil over the final binary. This flag is enabled by default since it's always been rustc's default behavior.
  • If disabled, the compiler both does not run dsymutil and it doesn't delete the object files on disk relevant for the final compilation. This ensures that references in the binary to the object files for debuginfo can be followed and found.

I'd like to, while opening this issue, also propose that this flag get stabilized. I think the interface is work bikeshedding (e.g. naming and such), but I think the functionality is all in what should be the final end state. I think the expected vision for how we ship this benefit to end users is to switch Cargo's defaults to passing run-dsymutil=no (or the equivalent) by default on macOS. That way macOS users should, by default, get the benefit of no dsymutil invocations but should retain the ability to debug executables as well as get filenames/line numbers with backtraces. Furthermore if *.dSYM is still required by the user, end-applications can always run it as part of a build process themselves.
@alexcrichton alexcrichton mentioned this issue Nov 23, 2020
Merged
@alexcrichton
Copy link
Member Author

It's also worth pointing out that this was spawned from the long-ago feature request of #47240

@camelid camelid added A-debuginfo Area: Debugging information in compiled programs (DWARF, PDB, etc.) A-runtime Area: std's runtime and "pre-main" init for handling backtraces, unwinds, stack overflows C-tracking-issue Category: An issue tracking the progress of sth. like the implementation of an RFC I-compiletime Issue: Problems and improvements with respect to compile times. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Nov 23, 2020
@nagisa
Copy link
Member

nagisa commented Nov 25, 2020

Replicating what I wrote on Zulip:

Given #77117, it seems to me like we want a flag that can encompass split debug info control in a generic flag, than having an ad-hoc flag for each of the architectures and targets we support. In particular -Zrun-dsymutil and -Zsplit-dwarf seem like they could be unified into a plain -Csplit-debuginfo with various values to control exact behaviour. On macOS I would expect -Zrun-dsymutil=no to become a -Csplit-debuginfo=off and -Zrun-dsymutil=yes to become a -Csplit-debuginfo=dsymutil or something along those lines.

@jryans jryans mentioned this issue Nov 28, 2020
Merged
@alexcrichton alexcrichton mentioned this issue Nov 30, 2020
Merged
@alexcrichton
Copy link
Member Author

Sorry meant to reply on Zulip, but I think that's a great suggestion! I've posted it as #79570

GuillaumeGomez added a commit to GuillaumeGomez/rust that referenced this issue Dec 1, 2020
@GuillaumeGomez
Rollup merge of rust-lang#79508 - jryans:check-dsymutil-result, r=nagisa
0206dd8 
Warn if `dsymutil` returns an error code

This checks the error code returned by `dsymutil` and warns if it failed. It
also provides the stdout and stderr logs from `dsymutil`, similar to the native
linker step.

I tried to think of ways to test this change, but so far I haven't found a good way, as you'd likely need to inject some nonsensical args into `dsymutil` to induce failure, which feels too artificial to me. Also, rust-lang#79361 suggests Rust is on the verge of disabling `dsymutil` by default, so perhaps it's okay for this change to be untested. In any case, I'm happy to add a test if someone sees a good approach.

Fixes rust-lang#78770
GuillaumeGomez added a commit to GuillaumeGomez/rust that referenced this issue Dec 1, 2020
@GuillaumeGomez
Rollup merge of rust-lang#79508 - jryans:check-dsymutil-result, r=nagisa
8b0c31d 
Warn if `dsymutil` returns an error code

This checks the error code returned by `dsymutil` and warns if it failed. It
also provides the stdout and stderr logs from `dsymutil`, similar to the native
linker step.

I tried to think of ways to test this change, but so far I haven't found a good way, as you'd likely need to inject some nonsensical args into `dsymutil` to induce failure, which feels too artificial to me. Also, rust-lang#79361 suggests Rust is on the verge of disabling `dsymutil` by default, so perhaps it's okay for this change to be untested. In any case, I'm happy to add a test if someone sees a good approach.

Fixes rust-lang#78770
alexcrichton added a commit to alexcrichton/rust that referenced this issue Jan 12, 2021
@alexcrichton
rustc: Stabilize -Zrun-dsymutil as -Csplit-debuginfo
f97c7a5 
This commit adds a new stable codegen option to rustc,
`-Csplit-debuginfo`. The old `-Zrun-dsymutil` flag is deleted and now
subsumed by this stable flag. Additionally `-Zsplit-dwarf` is also
subsumed by this flag but still requires `-Zunstable-options` to
actually activate. The `-Csplit-debuginfo` flag takes one of
three values:

* `off` - This indicates that split-debuginfo from the final artifact is
  not desired. This is not supported on Windows and is the default on
  Unix platforms except macOS. On macOS this means that `dsymutil` is
  not executed.

* `packed` - This means that debuginfo is desired in one location
  separate from the main executable. This is the default on Windows
  (`*.pdb`) and macOS (`*.dSYM`). On other Unix platforms this subsumes
  `-Zsplit-dwarf=single` and produces a `*.dwp` file.

* `unpacked` - This means that debuginfo will be roughly equivalent to
  object files, meaning that it's throughout the build directory
  rather than in one location (often the fastest for local development).
  This is not the default on any platform and is not supported on Windows.

Each target can indicate its own default preference for how debuginfo is
handled. Almost all platforms default to `off` except for Windows and
macOS which default to `packed` for historical reasons.

Some equivalencies for previous unstable flags with the new flags are:

* `-Zrun-dsymutil=yes` -> `-Csplit-debuginfo=packed`
* `-Zrun-dsymutil=no` -> `-Csplit-debuginfo=unpacked`
* `-Zsplit-dwarf=single` -> `-Csplit-debuginfo=packed`
* `-Zsplit-dwarf=split` -> `-Csplit-debuginfo=unpacked`

Note that `-Csplit-debuginfo` still requires `-Zunstable-options` for
non-macOS platforms since split-dwarf support was *just* implemented in
rustc.

There's some more rationale listed on rust-lang#79361, but the main gist of the
motivation for this commit is that `dsymutil` can take quite a long time
to execute in debug builds and provides little benefit. This means that
incremental compile times appear that much worse on macOS because the
compiler is constantly running `dsymutil` over every single binary it
produces during `cargo build` (even build scripts!). Ideally rustc would
switch to not running `dsymutil` by default, but that's a problem left
to get tackled another day.

Closes rust-lang#79361
alexcrichton added a commit to alexcrichton/rust that referenced this issue Jan 25, 2021
@alexcrichton
rustc: Stabilize -Zrun-dsymutil as -Csplit-debuginfo
b8852d4 
This commit adds a new stable codegen option to rustc,
`-Csplit-debuginfo`. The old `-Zrun-dsymutil` flag is deleted and now
subsumed by this stable flag. Additionally `-Zsplit-dwarf` is also
subsumed by this flag but still requires `-Zunstable-options` to
actually activate. The `-Csplit-debuginfo` flag takes one of
three values:

* `off` - This indicates that split-debuginfo from the final artifact is
  not desired. This is not supported on Windows and is the default on
  Unix platforms except macOS. On macOS this means that `dsymutil` is
  not executed.

* `packed` - This means that debuginfo is desired in one location
  separate from the main executable. This is the default on Windows
  (`*.pdb`) and macOS (`*.dSYM`). On other Unix platforms this subsumes
  `-Zsplit-dwarf=single` and produces a `*.dwp` file.

* `unpacked` - This means that debuginfo will be roughly equivalent to
  object files, meaning that it's throughout the build directory
  rather than in one location (often the fastest for local development).
  This is not the default on any platform and is not supported on Windows.

Each target can indicate its own default preference for how debuginfo is
handled. Almost all platforms default to `off` except for Windows and
macOS which default to `packed` for historical reasons.

Some equivalencies for previous unstable flags with the new flags are:

* `-Zrun-dsymutil=yes` -> `-Csplit-debuginfo=packed`
* `-Zrun-dsymutil=no` -> `-Csplit-debuginfo=unpacked`
* `-Zsplit-dwarf=single` -> `-Csplit-debuginfo=packed`
* `-Zsplit-dwarf=split` -> `-Csplit-debuginfo=unpacked`

Note that `-Csplit-debuginfo` still requires `-Zunstable-options` for
non-macOS platforms since split-dwarf support was *just* implemented in
rustc.

There's some more rationale listed on rust-lang#79361, but the main gist of the
motivation for this commit is that `dsymutil` can take quite a long time
to execute in debug builds and provides little benefit. This means that
incremental compile times appear that much worse on macOS because the
compiler is constantly running `dsymutil` over every single binary it
produces during `cargo build` (even build scripts!). Ideally rustc would
switch to not running `dsymutil` by default, but that's a problem left
to get tackled another day.

Closes rust-lang#79361
JohnTitor added a commit to JohnTitor/rust that referenced this issue Jan 26, 2021
@JohnTitor
Rollup merge of rust-lang#79570 - alexcrichton:split-debuginfo, r=bjorn3
4ace4e6 
rustc: Stabilize `-Zrun-dsymutil` as `-Csplit-debuginfo`

This commit adds a new stable codegen option to rustc,
`-Csplit-debuginfo`. The old `-Zrun-dsymutil` flag is deleted and now
subsumed by this stable flag. Additionally `-Zsplit-dwarf` is also
subsumed by this flag but still requires `-Zunstable-options` to
actually activate. The `-Csplit-debuginfo` flag takes one of
three values:

* `off` - This indicates that split-debuginfo from the final artifact is
  not desired. This is not supported on Windows and is the default on
  Unix platforms except macOS. On macOS this means that `dsymutil` is
  not executed.

* `packed` - This means that debuginfo is desired in one location
  separate from the main executable. This is the default on Windows
  (`*.pdb`) and macOS (`*.dSYM`). On other Unix platforms this subsumes
  `-Zsplit-dwarf=single` and produces a `*.dwp` file.

* `unpacked` - This means that debuginfo will be roughly equivalent to
  object files, meaning that it's throughout the build directory
  rather than in one location (often the fastest for local development).
  This is not the default on any platform and is not supported on Windows.

Each target can indicate its own default preference for how debuginfo is
handled. Almost all platforms default to `off` except for Windows and
macOS which default to `packed` for historical reasons.

Some equivalencies for previous unstable flags with the new flags are:

* `-Zrun-dsymutil=yes` -> `-Csplit-debuginfo=packed`
* `-Zrun-dsymutil=no` -> `-Csplit-debuginfo=unpacked`
* `-Zsplit-dwarf=single` -> `-Csplit-debuginfo=packed`
* `-Zsplit-dwarf=split` -> `-Csplit-debuginfo=unpacked`

Note that `-Csplit-debuginfo` still requires `-Zunstable-options` for
non-macOS platforms since split-dwarf support was *just* implemented in
rustc.

There's some more rationale listed on rust-lang#79361, but the main gist of the
motivation for this commit is that `dsymutil` can take quite a long time
to execute in debug builds and provides little benefit. This means that
incremental compile times appear that much worse on macOS because the
compiler is constantly running `dsymutil` over every single binary it
produces during `cargo build` (even build scripts!). Ideally rustc would
switch to not running `dsymutil` by default, but that's a problem left
to get tackled another day.

Closes rust-lang#79361
alexcrichton added a commit to alexcrichton/rust that referenced this issue Jan 26, 2021
@alexcrichton
rustc: Stabilize -Zrun-dsymutil as -Csplit-debuginfo
cdcf92f 
This commit adds a new stable codegen option to rustc,
`-Csplit-debuginfo`. The old `-Zrun-dsymutil` flag is deleted and now
subsumed by this stable flag. Additionally `-Zsplit-dwarf` is also
subsumed by this flag but still requires `-Zunstable-options` to
actually activate. The `-Csplit-debuginfo` flag takes one of
three values:

* `off` - This indicates that split-debuginfo from the final artifact is
  not desired. This is not supported on Windows and is the default on
  Unix platforms except macOS. On macOS this means that `dsymutil` is
  not executed.

* `packed` - This means that debuginfo is desired in one location
  separate from the main executable. This is the default on Windows
  (`*.pdb`) and macOS (`*.dSYM`). On other Unix platforms this subsumes
  `-Zsplit-dwarf=single` and produces a `*.dwp` file.

* `unpacked` - This means that debuginfo will be roughly equivalent to
  object files, meaning that it's throughout the build directory
  rather than in one location (often the fastest for local development).
  This is not the default on any platform and is not supported on Windows.

Each target can indicate its own default preference for how debuginfo is
handled. Almost all platforms default to `off` except for Windows and
macOS which default to `packed` for historical reasons.

Some equivalencies for previous unstable flags with the new flags are:

* `-Zrun-dsymutil=yes` -> `-Csplit-debuginfo=packed`
* `-Zrun-dsymutil=no` -> `-Csplit-debuginfo=unpacked`
* `-Zsplit-dwarf=single` -> `-Csplit-debuginfo=packed`
* `-Zsplit-dwarf=split` -> `-Csplit-debuginfo=unpacked`

Note that `-Csplit-debuginfo` still requires `-Zunstable-options` for
non-macOS platforms since split-dwarf support was *just* implemented in
rustc.

There's some more rationale listed on rust-lang#79361, but the main gist of the
motivation for this commit is that `dsymutil` can take quite a long time
to execute in debug builds and provides little benefit. This means that
incremental compile times appear that much worse on macOS because the
compiler is constantly running `dsymutil` over every single binary it
produces during `cargo build` (even build scripts!). Ideally rustc would
switch to not running `dsymutil` by default, but that's a problem left
to get tackled another day.

Closes rust-lang#79361
Dylan-DPC-zz pushed a commit to Dylan-DPC-zz/rust that referenced this issue Jan 27, 2021
@Dylan-DPC
Rollup merge of rust-lang#79570 - alexcrichton:split-debuginfo, r=bjorn3
a67e6f9 
rustc: Stabilize `-Zrun-dsymutil` as `-Csplit-debuginfo`

This commit adds a new stable codegen option to rustc,
`-Csplit-debuginfo`. The old `-Zrun-dsymutil` flag is deleted and now
subsumed by this stable flag. Additionally `-Zsplit-dwarf` is also
subsumed by this flag but still requires `-Zunstable-options` to
actually activate. The `-Csplit-debuginfo` flag takes one of
three values:

* `off` - This indicates that split-debuginfo from the final artifact is
  not desired. This is not supported on Windows and is the default on
  Unix platforms except macOS. On macOS this means that `dsymutil` is
  not executed.

* `packed` - This means that debuginfo is desired in one location
  separate from the main executable. This is the default on Windows
  (`*.pdb`) and macOS (`*.dSYM`). On other Unix platforms this subsumes
  `-Zsplit-dwarf=single` and produces a `*.dwp` file.

* `unpacked` - This means that debuginfo will be roughly equivalent to
  object files, meaning that it's throughout the build directory
  rather than in one location (often the fastest for local development).
  This is not the default on any platform and is not supported on Windows.

Each target can indicate its own default preference for how debuginfo is
handled. Almost all platforms default to `off` except for Windows and
macOS which default to `packed` for historical reasons.

Some equivalencies for previous unstable flags with the new flags are:

* `-Zrun-dsymutil=yes` -> `-Csplit-debuginfo=packed`
* `-Zrun-dsymutil=no` -> `-Csplit-debuginfo=unpacked`
* `-Zsplit-dwarf=single` -> `-Csplit-debuginfo=packed`
* `-Zsplit-dwarf=split` -> `-Csplit-debuginfo=unpacked`

Note that `-Csplit-debuginfo` still requires `-Zunstable-options` for
non-macOS platforms since split-dwarf support was *just* implemented in
rustc.

There's some more rationale listed on rust-lang#79361, but the main gist of the
motivation for this commit is that `dsymutil` can take quite a long time
to execute in debug builds and provides little benefit. This means that
incremental compile times appear that much worse on macOS because the
compiler is constantly running `dsymutil` over every single binary it
produces during `cargo build` (even build scripts!). Ideally rustc would
switch to not running `dsymutil` by default, but that's a problem left
to get tackled another day.

Closes rust-lang#79361
Dylan-DPC-zz pushed a commit to Dylan-DPC-zz/rust that referenced this issue Jan 27, 2021
@Dylan-DPC
Rollup merge of rust-lang#79570 - alexcrichton:split-debuginfo, r=bjorn3
2abb695 
rustc: Stabilize `-Zrun-dsymutil` as `-Csplit-debuginfo`

This commit adds a new stable codegen option to rustc,
`-Csplit-debuginfo`. The old `-Zrun-dsymutil` flag is deleted and now
subsumed by this stable flag. Additionally `-Zsplit-dwarf` is also
subsumed by this flag but still requires `-Zunstable-options` to
actually activate. The `-Csplit-debuginfo` flag takes one of
three values:

* `off` - This indicates that split-debuginfo from the final artifact is
  not desired. This is not supported on Windows and is the default on
  Unix platforms except macOS. On macOS this means that `dsymutil` is
  not executed.

* `packed` - This means that debuginfo is desired in one location
  separate from the main executable. This is the default on Windows
  (`*.pdb`) and macOS (`*.dSYM`). On other Unix platforms this subsumes
  `-Zsplit-dwarf=single` and produces a `*.dwp` file.

* `unpacked` - This means that debuginfo will be roughly equivalent to
  object files, meaning that it's throughout the build directory
  rather than in one location (often the fastest for local development).
  This is not the default on any platform and is not supported on Windows.

Each target can indicate its own default preference for how debuginfo is
handled. Almost all platforms default to `off` except for Windows and
macOS which default to `packed` for historical reasons.

Some equivalencies for previous unstable flags with the new flags are:

* `-Zrun-dsymutil=yes` -> `-Csplit-debuginfo=packed`
* `-Zrun-dsymutil=no` -> `-Csplit-debuginfo=unpacked`
* `-Zsplit-dwarf=single` -> `-Csplit-debuginfo=packed`
* `-Zsplit-dwarf=split` -> `-Csplit-debuginfo=unpacked`

Note that `-Csplit-debuginfo` still requires `-Zunstable-options` for
non-macOS platforms since split-dwarf support was *just* implemented in
rustc.

There's some more rationale listed on rust-lang#79361, but the main gist of the
motivation for this commit is that `dsymutil` can take quite a long time
to execute in debug builds and provides little benefit. This means that
incremental compile times appear that much worse on macOS because the
compiler is constantly running `dsymutil` over every single binary it
produces during `cargo build` (even build scripts!). Ideally rustc would
switch to not running `dsymutil` by default, but that's a problem left
to get tackled another day.

Closes rust-lang#79361
JohnTitor added a commit to JohnTitor/rust that referenced this issue Jan 29, 2021
@JohnTitor
Rollup merge of rust-lang#79570 - alexcrichton:split-debuginfo, r=bjorn3
d9e56f4 
rustc: Stabilize `-Zrun-dsymutil` as `-Csplit-debuginfo`

This commit adds a new stable codegen option to rustc,
`-Csplit-debuginfo`. The old `-Zrun-dsymutil` flag is deleted and now
subsumed by this stable flag. Additionally `-Zsplit-dwarf` is also
subsumed by this flag but still requires `-Zunstable-options` to
actually activate. The `-Csplit-debuginfo` flag takes one of
three values:

* `off` - This indicates that split-debuginfo from the final artifact is
  not desired. This is not supported on Windows and is the default on
  Unix platforms except macOS. On macOS this means that `dsymutil` is
  not executed.

* `packed` - This means that debuginfo is desired in one location
  separate from the main executable. This is the default on Windows
  (`*.pdb`) and macOS (`*.dSYM`). On other Unix platforms this subsumes
  `-Zsplit-dwarf=single` and produces a `*.dwp` file.

* `unpacked` - This means that debuginfo will be roughly equivalent to
  object files, meaning that it's throughout the build directory
  rather than in one location (often the fastest for local development).
  This is not the default on any platform and is not supported on Windows.

Each target can indicate its own default preference for how debuginfo is
handled. Almost all platforms default to `off` except for Windows and
macOS which default to `packed` for historical reasons.

Some equivalencies for previous unstable flags with the new flags are:

* `-Zrun-dsymutil=yes` -> `-Csplit-debuginfo=packed`
* `-Zrun-dsymutil=no` -> `-Csplit-debuginfo=unpacked`
* `-Zsplit-dwarf=single` -> `-Csplit-debuginfo=packed`
* `-Zsplit-dwarf=split` -> `-Csplit-debuginfo=unpacked`

Note that `-Csplit-debuginfo` still requires `-Zunstable-options` for
non-macOS platforms since split-dwarf support was *just* implemented in
rustc.

There's some more rationale listed on rust-lang#79361, but the main gist of the
motivation for this commit is that `dsymutil` can take quite a long time
to execute in debug builds and provides little benefit. This means that
incremental compile times appear that much worse on macOS because the
compiler is constantly running `dsymutil` over every single binary it
produces during `cargo build` (even build scripts!). Ideally rustc would
switch to not running `dsymutil` by default, but that's a problem left
to get tackled another day.

Closes rust-lang#79361
@bors bors closed this as completed in a124043 Jan 29, 2021
@JohanEngelen JohanEngelen mentioned this issue Jun 26, 2022
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-debuginfo Area: Debugging information in compiled programs (DWARF, PDB, etc.) A-runtime Area: std's runtime and "pre-main" init for handling backtraces, unwinds, stack overflows C-tracking-issue Category: An issue tracking the progress of sth. like the implementation of an RFC I-compiletime Issue: Problems and improvements with respect to compile times. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

rustc: Stabilize -Zrun-dsymutil as -Csplit-debuginfo alexcrichton/rust
3 participants
@alexcrichton @nagisa @camelid