Stabilize --extern flag without a path. #64882

ehuss · 2019-09-28T20:01:53Z

This stabilizes the --extern flag without a path, implemented in #54116.

This flag is used to add a crate that may be found in the search path to the extern prelude. The intent of stabilizing this now is to change Cargo to emit this flag for proc_macro when building a proc-macro crate. This will allow the ability to elide extern crate proc_macro; for proc-macros, one of the few places where it is still necessary.

It is intended that Cargo may also use this flag for other cases in the future as part of the std-aware work. There will likely be some kind of syntax where users may declare dependencies on other crates (such as alloc), and Cargo will use this flag so that they may be used like any other crate. At this time there are no short-term plans to use it for anything other than proc-macro.

This will not help for non-proc-macro crates that use proc_macro, which I believe is not too common?

An alternate approach for proc-macro is to use the meta crate, but from my inquiries there doesn't appear to be anyone interested in pushing that forward. The meta crate also doesn't help with things like alloc or test.

cc #57288

Centril · 2019-09-28T20:31:47Z

src/test/run-make-fulldeps/extern-flag-fun/Makefile

Is this the only test?

extern-flag-fun is not actually testing this feature. It is a 5-year old test from when --extern was first added. The removed --extern hello line was originally there to verify that it would error due to the missing path. When pathless --extern was added, the error changed to "missing -Z unstable-options". With this PR, the hello line no longer fails (and thus breaks the test), due to -L flags injected by the makefile. I figured it no longer fit the spirit of the test.

I have added some tests.

Centril · 2019-09-28T20:32:45Z

src/doc/rustc/src/command-line-arguments.md

Do we have tests ensuring that sysroot crates, e.g. private rustc details emit an error when used on a stable compiler? (Also when an explicit path is used to them, not just a search.)

What happens if both --extern CRATENAME=PATH and --extern CRATENAME are specified for the same CRATENAME? It would be nice to document that.
AFAIR, multiple --extern CRATENAME=PATHs with the same CRATENAME are possible if they lead to different kinds of libraries, like rlib and dylib.

There are about 30 tests doing various things with pathless --extern (search for // compile-flags:.*--extern).

I have added some more specific tests:

run-make-fulldeps/extern-flag-pathless: Shows what happens when you mix pathless with path --extern flags. Also demonstrates the preference of rlib over dylib.

ui-fulldeps/pathless-extern-unstable.rs: Shows --extern rustc is not allowed.

ui/pathless-extern-ok.rs: Basic test for a sysroot crate.

I don't have a good intuition of how sophisticated rustc tests should be. They are also a bit disorganized, so I didn't know where to stick them, or which style would be preferred. I think the dylib tests should be safe from a cross platform standpoint (I believe they only run on Linux?).

I updated the documentation with some more detail. I never really know how detailed rustc docs should be. I intentionally left the prefer-dynamic algorithm vague. It is described somewhat in src/librustc_metadata/dependency_format.rs. I figure since it is complex, and possibly subject to change, it may not be worthwhile. I can add more or remove some if desired.

withoutboats · 2019-10-03T19:43:10Z

We discussed this in the lang team meeting. In general, we are comfortable with stabilizing this feature. We would like some more commentary from other people with more knowledge of build systems, the sysroot, etc (lang team members with more relevant experience here weren't available today), mainly about what possible hazards there could be with this feature.

In terms of the larger goal this is trying to solve (eliminating the remaining use cases for extern crate), we had some other thoughts. This is really something that has been dropped on the floor, and no one seems available to shepherd it. It seems good to make proc_macro available in crates with proc-macro marked in lib, and this plus the associated change to cargo seems like a great solution to that problem. In the long term, we would like to reorganize the proc_macro APIs under meta and deprecate proc_macro, but - again - no one is shepherding that (as you commented).

However, it's not clear if this will be a step to solving the no_std and test problems, which have some important differences (most importantly, currently there's no clear way the need for these crates is marked in the toml). We think a full RFC process would be necessary to figure out the interface to access these crates without using extern crate.

ehuss · 2019-10-03T20:50:07Z

We think a full RFC process would be necessary to figure out the interface to access these crates without using extern crate.

Absolutely. That is the intent of rust-lang/wg-cargo-std-aware#5 to produce an RFC with the necessary Cargo.toml syntax to make these dependencies explicit, and is the next major milestone I plan to tackle.

Centril · 2019-10-06T20:55:17Z

src/test/ui/pathless-extern-ok.rs

Is this achieved by whitelist?

In a sense, yes. The crate must be marked with #![stable], otherwise the -Zforce-unstable-if-unmarked makes it unstable.

Ah... so just for my education, what happens if -Z force-unstable-if-unmarked isn't there? (And it cannot be there on stable when you are not using cargo since it's a -Z flag?)

It will successfully link whatever library is in the given path. That library will be added to the extern prelude with the identifier "alloc".

Isn't that a problem then if you are not using cargo since it would allow you to use unstable stuff on stable? I'm rather surprised that -Z force-unstable-if-unmarked isn't the default and that you need a flag to opt-out for sysroot crates.

Sorry, I'm not understanding the question.

On stable rustc, you cannot access any crates that are marked "unstable". Either from --extern foo or --extern foo=path/to/sysroot/libfoo.rlib or extern crate foo; or -L path/to/unstable/crates.

On nightly, you can include the attribute #![feature(rustc_private)] to override this check.

All of the sysroot crates are built with the -Zforce-unstable-if-unmarked flag.

The opt-in nature is just saying #![stable] crates are OK to access, even with the -Zforce-unstable-if-unmarked flag (that is the "if unmarked" part). I believe the only crates marked stable are core, std, alloc, and proc_macro.

In theory, anyone can compile any crate using unstable features using nightly, and then load those crates from stable. In practice, you can't (due to rustc version checking) or clearly something you shouldn't do (like set RUSTC_BOOTSTRAP env var).

Can you say more how you think that can be circumvented?

All of the sysroot crates are built with the -Zforce-unstable-if-unmarked flag.

Oh! -- I missed this part. I thought you meant that when the sysroot crate is used, unless there is a -Zforce..., then it won't be gated... but I see now that this is about when the crate is built, not used. Now it clicks. Maybe this conversation should be distilled into the rustc guide?

Done: rust-lang/rustc-dev-guide#462

src/test/ui-fulldeps/pathless-extern-unstable.rs

withoutboats · 2019-10-07T19:46:10Z

Absolutely. That is the intent of rust-lang/wg-cargo-std-aware#5 to produce an RFC with the necessary Cargo.toml syntax to make these dependencies explicit, and is the next major milestone I plan to tackle.

As a note, one reason I think we have not been enthusiastic on this is for the primary use case - core, alloc - there's a lot of good reason to deprecate these crates and move to a system that works more like feature flags on a single crate (so we can add impls without orphan issues, for example). However, this is a big and complicated refactor of the standard library and no one is driving it.

nikomatsakis · 2019-10-10T18:47:04Z

@rfcbot fcp merge

Dear compiler team,

I propose that we merge this PR. It adds a --extern foo flag which adds a crate foo from the syspath (in contrast to --extern foo=path which adds the crate foo from a specific path). The lang team has signed off on the general concept.

rfcbot · 2019-10-10T18:47:05Z

ehuss · 2019-10-15T21:17:55Z

I just realized that this does not support crate renaming (the equivalent of extern crate test as std_test). Some options:

Delay this stabilization until it is supported.
Add support later through some extension of the existing CLI syntax.
Don't support renaming.

It looks like it would be relatively easy to add. I'm curious what anyone else thinks. I'd also like to hear if anyone would have any ideas on the cli syntax.

I think I'd like to lean towards 1 or 2, as I think it would be an odd wart not to support it. And if others agree, maybe waiting would be prudent?

petrochenkov · 2019-10-15T21:42:46Z

2 is ok, IMO.

The semantics are clear

--extern foo=PATH // search by exact path `PATH` and add to extern prelude as `foo`
--extern foo???bar // search by name `bar` in the search directories and add to extern prelude as `foo`
--extern foo // sugar for `--extern foo???foo`

we only need to review the syntactic space so that --extern foo???bar is never ambiguous with --extern foo=PATH and perhaps with --extern:modifier foo[=PATH|???bar].

petrochenkov · 2019-10-16T08:30:44Z

src/doc/rustc/src/command-line-arguments.md

Do we want to guarantee that? Does any tools rely on that? (The "rlib ->rmeta -> dylib" order for metadata.)

Hypothetically, what if we want to change it to "rmeta -> rlib -> dylib"?
rmeta is smaller, easier to obtain, perhaps faster to read, etc - same reasons why rlib is preferred to dylib, basically. Perhaps we can exploit that somehow.
Then maybe it better stay unspecified in that case?

Sounds good, I have removed it. I am not aware of any tools relying on that behavior, and I think it would be a little strange to do so.

JohnCSimon · 2019-10-26T03:38:52Z

Ping from triage:
This PR seems to be waiting on review @frewsxcv @petrochenkov
Cc: @ehuss
Thanks

frewsxcv · 2019-10-26T11:19:35Z

I'm not familiar with any of this code; can someone else review it?

Centril · 2019-10-26T13:48:57Z

Ping @Zoxc @michaelwoerister, and @nagisa.

r? @eddyb or maybe @petrochenkov

Centril · 2019-11-07T13:24:42Z

^-- Failed in #66185 (comment), @bors r-

GuillaumeGomez · 2019-11-07T13:24:54Z

Please fix the id duplicate then it can get r+ again.

@bors: r-

Add ERROR Co-Authored-By: Mazdak Farrokhzad <twingoow@gmail.com>

ehuss · 2019-11-07T15:22:22Z

@bors r=eddyb

Strange that github/bors did not have a merge conflict, since I got one locally.

bors · 2019-11-07T15:22:24Z

📌 Commit ee459c6 has been approved by eddyb

bors · 2019-11-08T01:15:56Z

⌛ Testing commit ee459c6 with merge d257440...

Stabilize --extern flag without a path. This stabilizes the `--extern` flag without a path, implemented in #54116. This flag is used to add a crate that may be found in the search path to the extern prelude. The intent of stabilizing this now is to change Cargo to emit this flag for `proc_macro` when building a proc-macro crate. This will allow the ability to elide `extern crate proc_macro;` for proc-macros, one of the few places where it is still necessary. It is intended that Cargo may also use this flag for other cases in the future as part of the [std-aware work](https://github.com/rust-lang/wg-cargo-std-aware/). There will likely be some kind of syntax where users may declare dependencies on other crates (such as `alloc`), and Cargo will use this flag so that they may be used like any other crate. At this time there are no short-term plans to use it for anything other than proc-macro. This will not help for non-proc-macro crates that use `proc_macro`, which I believe is not too common? An alternate approach for proc-macro is to use the `meta` crate, but from my inquiries there doesn't appear to be anyone interested in pushing that forward. The `meta` crate also doesn't help with things like `alloc` or `test`. cc #57288

bors · 2019-11-08T04:21:52Z

☀️ Test successful - checks-azure
Approved by: eddyb
Pushing d257440 to master...

rust-highfive · 2019-11-08T04:23:11Z

📣 Toolstate changed by #64882!

Tested on commit d257440.
Direct link to PR: #64882

💔 rls on linux: test-pass → test-fail (cc @Xanewok, @rust-lang/infra).

@Xanewok

Tested on commit rust-lang/rust@d257440. Direct link to PR: <rust-lang/rust#64882> 💔 rls on linux: test-pass → test-fail (cc @Xanewok, @rust-lang/infra).

Add proc_macro to the extern prelude. This makes it so that a proc-macro library can use the `proc_macro` crate without the `extern crate proc_macro;` item on the 2018 edition. This is the Cargo half of rust-lang/rust#64882.

Version 1.41.0 (2020-01-30) =========================== Language -------- - [You can now pass type parameters to foreign items when implementing traits.][65879] E.g. You can now write `impl<T> From<Foo> for Vec<T> {}`. - [You can now arbitrarily nest receiver types in the `self` position.][64325] E.g. you can now write `fn foo(self: Box<Box<Self>>) {}`. Previously only `Self`, `&Self`, `&mut Self`, `Arc<Self>`, `Rc<Self>`, and `Box<Self>` were allowed. - [You can now use any valid identifier in a `format_args` macro.][66847] Previously identifiers starting with an underscore were not allowed. - [Visibility modifiers (e.g. `pub`) are now syntactically allowed on trait items and enum variants.][66183] These are still rejected semantically, but can be seen and parsed by procedural macros and conditional compilation. Compiler -------- - [Rustc will now warn if you have unused loop `'label`s.][66325] - [Removed support for the `i686-unknown-dragonfly` target.][67255] - [Added tier 3 support\* for the `riscv64gc-unknown-linux-gnu` target.][66661] - [You can now pass an arguments file passing the `@path` syntax to rustc.][66172] Note that the format differs somewhat from what is found in other tooling; please see [the documentation][argfile-docs] for more information. - [You can now provide `--extern` flag without a path, indicating that it is available from the search path or specified with an `-L` flag.][64882] \* Refer to Rust's [platform support page][forge-platform-support] for more information on Rust's tiered platform support. [argfile-docs]: https://doc.rust-lang.org/nightly/rustc/command-line-arguments.html#path-load-command-line-flags-from-a-path Libraries --------- - [The `core::panic` module is now stable.][66771] It was already stable through `std`. - [`NonZero*` numerics now implement `From<NonZero*>` if it's a smaller integer width.][66277] E.g. `NonZeroU16` now implements `From<NonZeroU8>`. - [`MaybeUninit<T>` now implements `fmt::Debug`.][65013] Stabilized APIs --------------- - [`Result::map_or`] - [`Result::map_or_else`] - [`std::rc::Weak::weak_count`] - [`std::rc::Weak::strong_count`] - [`std::sync::Weak::weak_count`] - [`std::sync::Weak::strong_count`] Cargo ----- - [Cargo will now document all the private items for binary crates by default.][cargo/7593] - [`cargo-install` will now reinstall the package if it detects that it is out of date.][cargo/7560] - [Cargo.lock now uses a more git friendly format that should help to reduce merge conflicts.][cargo/7579] - [You can now override specific dependencies's build settings][cargo/7591] E.g. `[profile.dev.overrides.image] opt-level = 2` sets the `image` crate's optimisation level to `2` for debug builds. You can also use `[profile.<profile>.build_overrides]` to override build scripts and their dependencies. Misc ---- - [You can now specify `edition` in documentation code blocks to compile the block for that edition.][66238] E.g. `edition2018` tells rustdoc that the code sample should be compiled the 2018 edition of Rust. - [You can now provide custom themes to rustdoc with `--theme`, and check the current theme with `--check-theme`.][54733] - [You can use `#[cfg(doc)]` to compile an item when building documentation.][61351] Compatibility Notes ------------------- - [As previously announced 1.41.0 will be the last tier 1 release for 32-bit Apple targets.][apple-32bit-drop] This means that the source code is still available to build, but the targets are no longer being tested and release binaries for those platforms will no longer be distributed by the Rust project. Please refer to the linked blog post for more information. [54733]: rust-lang/rust#54733 [61351]: rust-lang/rust#61351 [67255]: rust-lang/rust#67255 [66661]: rust-lang/rust#66661 [66771]: rust-lang/rust#66771 [66847]: rust-lang/rust#66847 [66238]: rust-lang/rust#66238 [66277]: rust-lang/rust#66277 [66325]: rust-lang/rust#66325 [66172]: rust-lang/rust#66172 [66183]: rust-lang/rust#66183 [65879]: rust-lang/rust#65879 [65013]: rust-lang/rust#65013 [64882]: rust-lang/rust#64882 [64325]: rust-lang/rust#64325 [cargo/7560]: rust-lang/cargo#7560 [cargo/7579]: rust-lang/cargo#7579 [cargo/7591]: rust-lang/cargo#7591 [cargo/7593]: rust-lang/cargo#7593 [`Result::map_or_else`]: https://doc.rust-lang.org/std/result/enum.Result.html#method.map_or_else [`Result::map_or`]: https://doc.rust-lang.org/std/result/enum.Result.html#method.map_or [`std::rc::Weak::weak_count`]: https://doc.rust-lang.org/std/rc/struct.Weak.html#method.weak_count [`std::rc::Weak::strong_count`]: https://doc.rust-lang.org/std/rc/struct.Weak.html#method.strong_count [`std::sync::Weak::weak_count`]: https://doc.rust-lang.org/std/sync/struct.Weak.html#method.weak_count [`std::sync::Weak::strong_count`]: https://doc.rust-lang.org/std/sync/struct.Weak.html#method.strong_count [apple-32bit-drop]: https://blog.rust-lang.org/2020/01/03/reducing-support-for-32-bit-apple-targets.html

rust-highfive assigned frewsxcv Sep 28, 2019

Centril added this to the 1.40 milestone Sep 28, 2019

Centril added the I-nominated label Sep 28, 2019

Centril reviewed Sep 28, 2019

View reviewed changes

Centril reviewed Oct 6, 2019

View reviewed changes

ehuss force-pushed the stabilize-bare-extern branch from c6ad892 to 6ba1de4 Compare October 8, 2019 16:45

ehuss mentioned this pull request Oct 9, 2019

Expand on stability documentation. rust-lang/rustc-dev-guide#462

Merged

nikomatsakis removed I-nominated T-lang Relevant to the language team labels Oct 10, 2019

rfcbot added proposed-final-comment-period Proposed to merge/close by relevant subteam, see T-<team> label. Will enter FCP once signed off. disposition-merge This issue / PR is in PFCP or FCP with a disposition to merge it. labels Oct 10, 2019

Centril removed the needs-fcp This change is insta-stable, or significant enough to need a team FCP to proceed. label Oct 11, 2019

petrochenkov reviewed Oct 16, 2019

View reviewed changes

ehuss force-pushed the stabilize-bare-extern branch from 6ba1de4 to 098ffd9 Compare October 16, 2019 16:55

JohnCSimon added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Oct 26, 2019

frewsxcv removed their assignment Oct 26, 2019

bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Nov 7, 2019

Centril modified the milestones: 1.40, 1.41 Nov 7, 2019

ehuss and others added 8 commits November 7, 2019 05:51

Stabilize --extern flag without a path.

b47e3d8

Add more --extern tests.

e0058ce

Update built-in help for --extern.

4bf411e

Update extern linking documentation.

b54e8ec

Update src/test/ui-fulldeps/pathless-extern-unstable.rs

41e051d

Add ERROR Co-Authored-By: Mazdak Farrokhzad <twingoow@gmail.com>

Add test for --extern alloc=librustc.rlib

f9e4f0f

Remove docs on --extern metadata precedence.

845ec5d

Update for unstable option refactoring.

ee459c6

ehuss force-pushed the stabilize-bare-extern branch from 098ffd9 to ee459c6 Compare November 7, 2019 15:21

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Nov 7, 2019

bors added the merged-by-bors This PR was explicitly merged by bors. label Nov 8, 2019

bors merged commit ee459c6 into rust-lang:master Nov 8, 2019

ehuss mentioned this pull request Dec 12, 2019

Add proc_macro to the extern prelude. rust-lang/cargo#7700

Merged

petrochenkov mentioned this pull request Jan 12, 2020

rustbuild: Both stage0 and stage1 std are rebuilt spuriously #68149

Closed

Stabilize --extern flag without a path. #64882

Stabilize --extern flag without a path. #64882

Uh oh!

Conversation

ehuss commented Sep 28, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ehuss Sep 30, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

withoutboats commented Oct 3, 2019

Uh oh!

ehuss commented Oct 3, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

withoutboats commented Oct 7, 2019

Uh oh!

nikomatsakis commented Oct 10, 2019

Uh oh!

rfcbot commented Oct 10, 2019 • edited by michaelwoerister Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ehuss commented Oct 15, 2019

Uh oh!

petrochenkov commented Oct 15, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JohnCSimon commented Oct 26, 2019

Uh oh!

frewsxcv commented Oct 26, 2019

Uh oh!

Centril commented Oct 26, 2019

Uh oh!

Centril commented Nov 7, 2019

Uh oh!

GuillaumeGomez commented Nov 7, 2019

Uh oh!

ehuss commented Nov 7, 2019

Uh oh!

bors commented Nov 7, 2019

Uh oh!

bors commented Nov 8, 2019

Uh oh!

bors commented Nov 8, 2019

Uh oh!

rust-highfive commented Nov 8, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

ehuss Sep 30, 2019 •

edited

Loading

rfcbot commented Oct 10, 2019 •

edited by michaelwoerister

Loading