Remove gensym in format_args

[matthewjasper]

This also fixes some things to allow us to export opaque macros from libcore:

• Don't consider items that are only reachable through opaque macros as public/exported (so they aren't linted as needing docs)
• Mark private items reachable from the root of libcore as unstable - they are now reachable (in principle) in other crates via macros in libcore

r? @petrochenkov

[petrochenkov]

Oh, this is interesting.
This is a start/part of a larger work on converting built-in macros from semi-transparent to opaque.

It would be good to implement some pre-requisites before doing that.
I'm really bad at creating issues for this kind of stuff, so it always ends up as one-line notes in my private todo list.

• First, the unstable annotations on private items you had to add.
  I wanted to avoid that.
  We already have a type privacy system that ensures that non-pub entities cannot be used from other crates in any "non-ephemeral" ways (affecting codegen or linkage).

  So, I think that we can raise reachability only for pub items in EmbargoVisitor::visit_macro_def.
  Any other private entities that can be reached through an opaque macro are "ephemeral" (modules, imports) and not marking them as reachable should have zero effect on other crates in theory.

[petrochenkov]

• Second, built-in macros are currently AST-based (even if it's not necessary for most of them) and are built using syntax::ext::build::AstBuilder, which is notoriously bad at keeping spans.
  It often just drops the ctxt part of the spans passed to it and uses SyntaxContext::empty() instead, losing all the hygiene, opaque or not.

  So, I thought, perhaps instead of fixing that obsolete piece of code, which AstBuilder is, perhaps we need to clone the proc_macro::quote macro into the compiler, modify it to produce syntax::tokenstream tokens stream rather than proc_macro token streams and use it to implement the built-in macros migrating to the token-based model simultaneously with opaque hygiene.

  That approach is also not without problems, e.g. we still don't have a token-based equivalent of ExprKind::Err for good error recovery.
  So it may be better to fix the parts of AstBuilder used by the converted macros, as a short-term measure.

[petrochenkov]

So, I think the problem with #![unstable] should be solved before we start converting macros from semi-transparent to opaque, but the problems with AstBuilder can be fixed in one-by-one fashion if they arise (#63146 fixes one of them, btw).

[petrochenkov]

Otherwise, LGTM.

[matthewjasper]

OK, so I've have a local rework of the handling of reachability through macros and ran into this case:

#![feature(decl_macro)]

mod n {
    pub struct B(pub(crate) p::C);
    impl B {
        pub fn new() -> Self {
            B(p::C)
        }
    }
    mod p {
        pub struct C;

        impl C {
            pub fn foo(&self) -> i32 {
                33
            }
        }
    }
}

// This macro can (currently) be called cross-crate
pub macro m() {
    n::B::new().0.foo()
}

I'm currently handling this by making the field of B reachable, but this would also make the fields of RangeInclusive in libcore reachable, and need stability attributes. The alternative would be to change field privacy to be unhygienic.

[petrochenkov]

Fields are also "ephemeral", but can link to non-ephemeral things like new reachable types with public methods.

The situation is pretty similar to modules, I think.

mod m {
    pub struct Z;
    impl Z {
        pub fn public() {}
    }
}
pub struct S {
    field: m::Z,
}

Both module m and field f are private and therefore not "reachable themselves", but they are reachable in the sense that their "contents" need to be visited and marked as reachable.
Is it possible to visit fields' and modules' contents to mark them reachable without marking the fields and modules themselves as reachable?

(An alternative would be to use two flags like "reachable for propagation" and "reachable for checking" or something.)

[petrochenkov]

Actually, it may be possible to just calculate reachability in two passes - first reach everything ignoring nominal visibilities like it's done now, then visit everything again and reset anything with non-pub nominal visibility to unreachable.
That leaves impls though, which don't have nominal visibility, so we'll have to recalculate impl reachability, which in its turn may render something else unreachable, leading to the conclusion that one (iterative) pass probably was a better idea.
Hmm.

[petrochenkov]

@matthewjasper
Is this ready? (Looks ready, but the label is still "waiting-on-author".)

[matthewjasper]

Yes, it's ready.

[petrochenkov]

@bors r+

Sorry for the delay, I wanted to write some comments, but all of them are non-blocking, so I'll just write them later.

[bors]

📌 Commit d9d9246 has been approved by petrochenkov

[petrochenkov]

Statics are actually protected by a kind of extension to type privacy, since they are real items existing during linking:

rust/src/librustc_privacy/lib.rs

Lines 1107 to 1110 in 813a3a5

	// Additionally, until better reachability analysis for macros 2.0 is available,
	// we prohibit access to private statics from other crates, this allows to give
	// more code internal visibility at link time. (Access to private functions
	// is already prohibited by type privacy for function types.)

[petrochenkov]

Other things in this list are also "ephemeral" and we need to "reach through" them rather than mark them as reachable themselves.
This optimization probably not too important though (no examples in libcore) and it would be better to somehow introduce the distinction between "reach including" and "reach excluding" nodes in the reachability visitor in general rather than in its macro part only.

[petrochenkov]

So, the consequence is that we don't mark Priv as reachable in situations like this:

pub fn foo()  where Priv: Condition { ... }

This case is not covered by type privacy and foo can be used from other crates.
Is Priv required to be reachable at link time in this case? I think no, so what this PR does is ok.
Anyway, if something breaks we'll get bug reports and add them into the test suite.