Panic inside panic when procedural macro is called with proc_macro::bridge::client #60593

fedochet · 2019-05-06T22:56:40Z

I am experiencing a strange panic that happens inside procedural macros when they are compiled as dynamic libraries, dynamically linked and then called explicitly using proc_macro::bridge::client.

It's quite hard to reproduce in small amount of code, so I've made up an example project where you can launch the tests and see the panic, it is here. Specifically, the stack backtrace can be seen here.

I've experimented with different compiler versions, and it seems that the only thing that triggers such panic is version alignment between compiler that compiled procedural macro itself, and compiler that compiled the code that is calling run method. As long as those compilers have different versions, everything works as expected.

I wasn't able to find the reason of this problem myself, so I thought it may be useful for you to know about this behaviour (maybe it is a bug of some sort).

The text was updated successfully, but these errors were encountered:

eddyb · 2019-05-06T23:21:15Z

First off, you have both of these in the stack trace:

src/libproc_macro/bridge/client.rs
/rustc/a3404557c54ea48fb8efc805d93c450beb3364d4/src/libproc_macro/bridge/server.rs

Did you build the former yourself? If not, how come it doesn't have the hash in it?
To be clear, you need that source to be from a340455, and the build settings to match the upstream, otherwise it won't be compatible.

I'm suspecting you have a libproc_macro laying around from 597f432 (which is the other hash).
Or maybe I can't trust the paths, so this doesn't mean anything.

One thing you could try is to have a proc macro crate as a dependency of your project, which will ensure it gets compiled with the same compiler and against the same libproc_macro, and then you can search in target/debug/deps for the .so.

Proc macros presumably work on nightly, and officially your usecase is not supported, so I can't spend a lot of time looking into it, sorry, but I'll try to when I can.

fedochet · 2019-05-07T14:34:24Z

@eddyb thanks for your time and efforts!

If I understand your question correctly, then no, I did not build client.rs myself, or at least I think so. I've used a stock nightly compiler, not locally compiled

However, I'm using #![feature(proc_macro_internals)] to be able to use extern crate proc_macro and its internals. Maybe this is what actually triggers compilation of this module, and this is why there are different paths in the stack trace (probably, part of it is from code that I've compiled, and part of it is from already compiled code that is used in procedural macro)

I don't get the part about making proc_macro a dependency. I wasn't able to add it to the Cargo.toml, or locate it on the crates.io. Can you please explain what did you mean?

eddyb · 2019-05-07T15:58:56Z

Not proc_macro the crate, but the proc macro crate you are building during the test, you can have it in the same repository, and use a path dependency, instead of creating it in a temporary directory.

fedochet · 2019-05-07T17:08:20Z

I've tried to do that, unfortunately with no luck

I've updated the repo with test that uses path dependency (you should cargo clean everything after test so there are no chance that old compiled macro is used)

eddyb · 2019-05-13T17:04:20Z

@fedochet "no luck" as in "it still causes the panic"? Or does it work as intended in that configuration?

fedochet · 2019-05-13T18:54:33Z

@eddyb yes, still panic inside panic with same stacktrace

eddyb · 2019-05-14T17:12:49Z

Alright, so a potential reason for rustc not hitting this is always using a different copy of the library, but with a compatible ABI.
Still, I can't think of any reason sharing libproc_macro.rlib between the server and the client wouldn't work, they shouldn't have any global resources in common.

edwin0cheng · 2019-05-28T15:33:05Z

Just copy from the project issue i wrote:

After a little bit of digging, my guess is, it is not related to ABI compatible problem. It seem to be related to the different of thread_local storage of static linkage and dynamic linkage.

The actual panic is this line.
And state is actually a thread_local :

rust/src/libproc_macro/bridge/client.rs

Line 269 in dc6db14

static BRIDGE_STATE: scoped_cell::ScopedCell<BridgeStateL> =

On the other hand, in the example project:

https://github.com/fedochet/rust-proc-macro-panic-inside-panic-expample/blob/6ef3a6452c19ab6edab1829d048368ce811b07c3/src/main.rs#L164

client.run function will call Bridge::enter and set the state to BridgeState::Connected. Note that this is statically link to main crate.

And after that, when the actual TokenStream start to parse, Bridge::with will be called from shared library crate (which is dynamic link) and it will trigger the panic.

It seem to be two thread local are different, such that the panic get raised. But I don't know enough about linux shared library and thread local mechanism to understand this bug.

eddyb · 2019-05-28T15:57:04Z

@edwin0cheng Static vs dynamic linking does sound scary.
However, note that Bridge::enter is called from this function:

rust/src/libproc_macro/bridge/client.rs

Lines 341 to 351 in 7da1185

 // FIXME(#53451) public to work around `Cannot create local mono-item` ICE, 

 // affecting not only the function itself, but also the `BridgeState` `thread_local!`. 

 pub extern "C" fn __run_expand1( 

 mut bridge: Bridge<'_>, 

 f: fn(crate::TokenStream) -> crate::TokenStream, 

 ) -> Buffer<u8> { 

 // The initial `cached_buffer` contains the input. 

 let mut b = bridge.cached_buffer.take(); 

 panic::catch_unwind(panic::AssertUnwindSafe(|| { 

 bridge.enter(|| {

Which is used as a fn pointer as part of the Client struct you get from the dylib.

To investigate further, you could transmute the Client, just before calling .run on it, to [*const (); 3], and print it (with {:?}), and also print proc_macro::bridge::client::__run_expand1 as *const () - if the latter is equal to the second (middle) element of the former array, then the dynamic loading of proc macro dylib picked up a statically linked fn (which would be bad if the TLS isn't also deduplicated).

edwin0cheng · 2019-05-30T15:16:07Z

@eddyb Thanks for the information, it helps a lot for me to understand how it works.

I seem to understand why this bug happens, would you mind to tell me does it make sense ?

I can confirmed that the thread_local i mentioned previously are different in Bridge::enter and Bridge::with (by inserting dbg! in rustc source code directly). And it is because Bridge::with use a wrong thread local coming from static linkage.

Why the static linkage is introduced ? Surprisingly it is introduced by syn crate we used to parse the input string to TokenStream. Seem like dynamic linker is "smart" enough to merge some code path from dynamic linkage to static linkage. And that's why it depends on how dynamic linker implementation.

But in rustc it doesn't use procmacro::TokenStream, it uses syntax::TokenStream directly :

rust/src/libsyntax_ext/proc_macro_impl.rs

Line 17 in 9ebf478

impl base::AttrProcMacro for AttrProcMacro {

use syntax::tokenstream::TokenStream;

// ....

impl base::AttrProcMacro for AttrProcMacro {
    fn expand<'cx>(&self,
                   ecx: &'cx mut ExtCtxt<'_>,
                   span: Span,
                   annotation: TokenStream,
                   annotated: TokenStream)
                   -> TokenStream {
        let server = proc_macro_server::Rustc::new(ecx);
        match self.client.run(&EXEC_STRATEGY, server, annotation, annotated) {
    // ....
}

And that's why in server.rs, it use a Server trait :

impl client::Client<fn(crate::TokenStream) -> crate::TokenStream> {
pub fn run<S: Server>(

So that it won't introduce the static linkage of thread_local in their code.

eddyb · 2019-05-30T21:53:15Z

There is a dynamic indirection between the Client::run and all the code that uses Bridge::{with,enter}. I don't think what you're saying is possible.

Can you do what I said? Print those transmuted pointers?
Without that information I don't see how we can do better than guess.

edwin0cheng · 2019-05-31T01:55:07Z

@eddyb Sure :

running 1 test
[
    0x00007fdd79d6b730,
    0x00007fdd79d6b290,
    0x00007fdd79d6a320,
]
0x00007fdd7bb6a7d8
[src/libproc_macro/bridge/client.rs:249] "$method" = "$method"
[src/libproc_macro/bridge/client.rs:319] p_with = 0x00007fdd7be330a0
[src/libproc_macro/bridge/client.rs:356] "__run_expand1" = "__run_expand1"
[src/libproc_macro/bridge/client.rs:295] p_enter = 0x00007fdd7a09fd00
[src/libproc_macro/bridge/client.rs:319] p_with = 0x00007fdd7a09fd00
[src/libproc_macro/bridge/client.rs:249] "$method" = "$method"
[src/libproc_macro/bridge/client.rs:319] p_with = 0x00007fdd7be330a0
thread 'test_getset_expansion' panicked at 'procedural macro API is used outside of a procedural macro', src/libproc_macro/bridge/client.rs:324:17
stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at /home/edwin/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.25/src/backtrace/libunwind.rs:97
   1: backtrace::backtrace::trace_unsynchronized
             at /home/edwin/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.25/src/backtrace/mod.rs:66
   2: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:47

The code I insert println which is in callsite of @fedochet test project:

   for proc_macro in *proc_macros {
        match proc_macro {
            ProcMacro::Bang { client, .. } => {
                unsafe {
                    let p1: &[*const (); 3] = std::mem::transmute(client);
                    let f = &proc_macro::bridge::client::__run_expand1;
                    println!("{:#?}", p1);
                    println!("{:#?}", f as *const _);
                }
                let result = client.run(
                    &EXEC_STRATEGY,
                    rustc_server::Rustc::default(),
                    parse_string("struct S{}").expect("Cannot parse code"),
                );

It is different. However, the other dbg! I insert in the rustc tell the different story. What i did is inserting dbg! in src/libproc_macro/bridge/client.rs:

impl Bridge<'_> {                                                                            
    fn enter<R>(self, f: impl FnOnce() -> R) -> R {                                          
       let p_enter = &BRIDGE_STATE as *const _;               // <-------------                               
        dbg!(p_enter);                                              // <-------------                               
                                                                                                                                                                                          
        // Hide the default panic output within `proc_macro` expansions.                     
        // NB. the server can't do this because it may use a different libstd.               
        static HIDE_PANICS_DURING_EXPANSION: Once = Once::new();                             
        HIDE_PANICS_DURING_EXPANSION.call_once(|| {                                          
            let prev = panic::take_hook();                                                   
            panic::set_hook(Box::new(move |info| {                                           
                let hide = BridgeState::with(|state| match state {                           
                    BridgeState::NotConnected => false,                                      
                    BridgeState::Connected(_) | BridgeState::InUse => true,                  
                });                                                                          
                if !hide {                                                                   
                    prev(info)                                                               
                }                                                                            
            }));                                                                             
        });                                                                                  
                                                                                             
        BRIDGE_STATE.with(|state| state.set(BridgeState::Connected(self), f))                
    }                                                                                        
                                                                                             
    fn with<R>(f: impl FnOnce(&mut Bridge<'_>) -> R) -> R {                                  
        let p_with = &BRIDGE_STATE as *const _;             // <-------------                                                                
        dbg!(p_with);                                                 // <-------------                                                 
                                                                                             
                                                                                             
        BridgeState::with(|state| match state {                                              
            BridgeState::NotConnected => {                                                   
                panic!("procedural macro API is used outside of a procedural macro");        
            }                                                                                
            BridgeState::InUse => {                                                          
                panic!("procedural macro API is used while it's already in use");            
            }                                                                                
            BridgeState::Connected(bridge) => f(bridge),                                     
        })                                                                                   
    }                                                                                        
}

edwin0cheng · 2019-05-31T03:42:56Z

@eddyb and @fedochet here is a minimal project which demonstrable the override effect.

eddyb · 2019-06-02T08:28:59Z

BRIDGE_STATE is just a wrapper constant, you need to use BRIDGE_STATE.with(|s| s as *const _)

edwin0cheng · 2019-06-02T09:05:58Z

Here is the output of BRIDGE_STATE.with(|s| s as *const _):

running 1 test
[
    0x00007fa1b5382280,
    0x00007fa1b5381de0,
    0x00007fa1b53812f0,
]
0x00007fa1b6f68d78
[src/libproc_macro/bridge/client.rs:249] "$method" = "$method"
[src/libproc_macro/bridge/client.rs:321] p_with = 0x00007fa1b58b0600
[src/libproc_macro/bridge/client.rs:359] "__run_expand1" = "__run_expand1"
[src/libproc_macro/bridge/client.rs:296] p_enter = 0x00007fa1b0006500
[src/libproc_macro/bridge/client.rs:321] p_with = 0x00007fa1b0006500
[src/libproc_macro/bridge/client.rs:249] "$method" = "$method"
[src/libproc_macro/bridge/client.rs:321] p_with = 0x00007fa1b58b0600

impl Bridge<'_> {                                                                         
    fn enter<R>(self, f: impl FnOnce() -> R) -> R {                                       
        BRIDGE_STATE.with(|s| {                                                           
            let p_enter = s as *const _;                                                  
            dbg!(p_enter);                                                                
        });                                                                               
                                                                                          
        // Hide the default panic output within `proc_macro` expansions.                  
        // NB. the server can't do this because it may use a different libstd.            
        static HIDE_PANICS_DURING_EXPANSION: Once = Once::new();                          
        HIDE_PANICS_DURING_EXPANSION.call_once(|| {                                       
            let prev = panic::take_hook();                                                
            panic::set_hook(Box::new(move |info| {                                        
                let hide = BridgeState::with(|state| match state {                        
                    BridgeState::NotConnected => false,                                   
                    BridgeState::Connected(_) | BridgeState::InUse => true,               
                });                                                                       
                if !hide {                                                                
                    prev(info)                                                            
                }                                                                         
            }));                                                                          
        });                                                                               
                                                                                          
        BRIDGE_STATE.with(|state| state.set(BridgeState::Connected(self), f))             
    }                                                                                     
                                                                                          
    fn with<R>(f: impl FnOnce(&mut Bridge<'_>) -> R) -> R {                               
        BRIDGE_STATE.with(|s| {                                                           
            let p_with = s as *const _;                                                   
            dbg!(p_with);                                                                 
        });                                                                               
                                                                                          
                                                                                          
        BridgeState::with(|state| match state {                                           
            BridgeState::NotConnected => {                                                
                panic!("procedural macro API is used outside of a procedural macro");     
            }                                                                             
            BridgeState::InUse => {                                                       
                panic!("procedural macro API is used while it's already in use");         
            }                                                                             
            BridgeState::Connected(bridge) => f(bridge),                                  
        })                                                                                
    }                                                                                     
}

Anyway, seem like this problem can be solved by adding a flag RTLD_DEEPBIND when loading the dynamic library in unix system in his usage.

https://stackoverflow.com/questions/34073051/when-we-are-supposed-to-use-rtld-deepbind

fedochet · 2019-06-02T13:25:03Z

I can confirm that RTLD_DEEPBIND indeed resolved the issue for me, thanks @edwin0cheng 🎉

Maybe the whole problem wasn't noticed in the compiler itself because the compiler and the procedural macros are compiled by different compilers after all? I mean that the final stage of the compiler is compiled with the previous stage compiler, and the macros is compiled with the final stage compiler and then is linked into it

In my example, the panic occurs only when the same compiler is used for the procedural macro and for the linking code compilation (and without RTLD_DEEPBIND). Maybe if we use the same compiler to compile the compiler itself and procedural macro, and then will use this new compiler and procedural macro together, we will see the same behaviour?

eddyb · 2019-06-02T15:05:51Z

@edwin0cheng Okay that's definitely not supposed to happen, assuming all of those calls are coming from inside the same proc macro .so.
Are all the p_with/p_enter pointers equal when RTLD_DEEPBIND is set?
If so, I don't see how using different compilers would help.
Different compilers might cause more duplication, not less.
(But this is already weird enough that it could be five bugs in a trenchcoat)

edwin0cheng · 2019-06-02T16:08:36Z

@edwin0cheng Okay that's definitely not supposed to happen, assuming all of those calls are coming from inside the same proc macro .so.
Are all the p_with/p_enter pointers equal when RTLD_DEEPBIND is set?

I think there are some misunderstandings about the usage of @fedochet project here. Let me try to explain more.
Basically , what his project want to do is a "cargo expand" without invoking rustc or cargo (It is very useful in rust ide development). The mechanism of his idea is following:

Using syn crate to parse an input string to libprocmacro::TokenStream. (Note that syn crate is just a wrapper of libprocmacro.)
Loading the target proc-macro crate dylib manually
Using the libprocmacro::bridge compiler api to expand the parsed token stream.
Convert the output tokenstream back to string.

And here is the output dbg! of the test:

[src/libproc_macro/bridge/client.rs:249] "$method" = "$method"
[src/libproc_macro/bridge/client.rs:321] p_with = 0x00007fa1b58b0600
[src/libproc_macro/bridge/client.rs:359] "__run_expand1" = "__run_expand1"
[src/libproc_macro/bridge/client.rs:296] p_enter = 0x00007fa1b0006500
[src/libproc_macro/bridge/client.rs:321] p_with = 0x00007fa1b0006500
[src/libproc_macro/bridge/client.rs:249] "$method" = "$method"
[src/libproc_macro/bridge/client.rs:321] p_with = 0x00007fa1b58b0600

Note that the first and second line is coming from step 1(syn::parse_str(code)) which is statically linked with the main program. As I said in previous post, in some unix environment, The dlopen without RTLD_DEEPBIND flag make dynamic linker looking up symbol in the dependencies of main executable first, before looking in the dylib it self. That's why the address of last log is same with the first one.

Different compilers might cause more duplication, not less.

It is true. But different compilers helps here since the relocation address are different, the dynamic linker will treat as they are different symbol, such that the searches result of symbols will be different.

eddyb · 2019-06-02T20:45:19Z

Technically proc_macro should continue to work even if deduplication happens, as long as each translation unit has a consistent view of TLS.
i still don't understand whether the problem is duplication or deduplication, and which logs jave the flag that fixes things set.
I suppose I will have to try to reproduce myself tomorrow, with an out-of-tree copy of `libproc_macro, and hopefully reduce it down to a self-contained test with no proc-macro code.
And then we can maybe figure out whose fault it is (worried the answer is "dynamic loading is just broken by default").

eddyb · 2019-06-03T15:47:34Z

@fedochet @edwin0cheng I can't reproduce rust-proc-macro-panic-inside-panic-expample, on:

rustc 1.37.0-nightly (7840a0b75 2019-05-31)
rustc 1.37.0-nightly (627486af1 2019-06-02)
rustc 1.36.0-nightly (d628c2e64 2019-05-05)

That last one is nightly-2019-05-06 which is explicitly mentioned.

I'm curious, what's your glibc version? Mine is 2.27.

fedochet · 2019-06-03T15:53:19Z

@eddyb mine is 2.23 according to ldd --version

roman-ubuntu@HP-ENVY-Notebook:~$ ldd --version
ldd (Ubuntu GLIBC 2.23-0ubuntu11) 2.23
Copyright (C) 2016 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

eddyb · 2019-06-03T17:53:02Z

@fedochet Any chance you could update? I've just asked some other people with Ubuntu installs and they can reproduce with 2.23 but not 2.27. I'm starting to strongly suspect an rtld bug that got fixed.
EDIT: 2.24 also reproduces, and 2.27 still doesn't (even on Ubuntu).

fedochet · 2019-06-03T19:11:22Z

@eddyb I've just tried 2.27 with ubuntu:18.04 docker image and it seems to work perfectly, the test in panic-inside-panic is passing

I guess that to prevent dependency from (possibly) broken version of libc we may use static linkage with something like musl?

eddyb · 2019-06-03T20:14:54Z

I was finally able to reproduce, by running this:

NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs-channels/archive/nixos-18.03.tar.gz \
nix-shell --pure -p rustup --run '\
    export set RUSTUP_HOME=$(pwd)/rustup-glibc-2.26; \
    rustup default nightly && \
    cargo clean && \
    cargo test -- --nocapture'

So that's using glibc-2.26-131, which means it's likely that 2.27 fixed this.

eddyb · 2019-06-03T21:19:32Z

Uhm, this looks like a drive-by bugfix: bminor/glibc@9d7a374#diff-249add90a52e252f250ab509ffbcf72fL125.
EDIT: I tried reverting it but I can't get the build to fail - I guess I need to start bisecting glibc?

fedochet · 2019-06-04T15:31:59Z

@eddyb I am not sure it is worth your time - we already have two options for fixing this (using DEEPBIND and upgrading to the latest glibc), and I believe that this bug is actually caused by glibc

Thank you and @edwin0cheng for digging into this! It would have taken me forever to pinpoint where the problem actually is. Maybe I will try to narrow it to specific line in glibc later on

eddyb · 2019-06-05T08:49:45Z

I found another thing that makes this bug go away (on glibc-2.26): manually creating the cdylib that a proc-macro is supposed to be - but it does trigger with the dylib crate type!
A proc-macro crate is somewhere in between, but I'm surprised to see how much it is like a dylib crate.

I've also narrowed it down to a very simple test.

I can finally see what's happening: syn, uses proc-macro2, which calls into proc_macro.
Very importantly, this doesn't work, in the strictest sense. It's attempted to check whether it works, and the corresponding panic is silenced, but it has a side-effect.

That side-effect happens to be "defining" Bridge::with for the main executable, in the dynamic linker, with the body using the specific static that is linked into the main executable.
And in some circumstances (glibc before 2.27, or some other yet-to-be-determined part of the toolchain being before a certain version, RTLD_DEEPBIND not being enabled, and using the dylib or proc-macro crate type, not cdylib), that "definition" is used to resolve an intra-translation-unit call in the .so that gets dynamically loaded.

So Bridge::with in the proc macro .so becomes the same as in the main executable... but Bridge::enter doesn't (since it was never called in the main executable), breaking that TU permanently.
Now, this form of late binding would be fine if it applied to the static as well, but it doesn't.

Presumably this behavior got changed because it introduces unsound inconsistencies into TUs?
I'm not sure the dylib crate type is actually broken on our end, it might just be that it can't make enough symbols as private as cdylib can, resulting in the unsoundly inconsistent behavior from the rtld.

EDIT: I should also say that this cannot possibly happen within rustc, unless it expands a proc_macro::quote!(...) invocation (which is unstable) before a real proc macro.

eddyb · 2019-06-05T08:59:52Z

I just tried changing RELEASE in that test - turns out that 1.30.1 doesn't repro but 1.31.0 does. I'll go bisect this a bit further (down to a specific nightly or even PR) and open a separate issue.

EDIT: also, I just noticed that ld version also differs between the two NixOS versions I was using, so maybe the issue is with that older ld version?

edwin0cheng · 2019-06-05T09:00:33Z

I can finally see what's happening: syn, uses proc-macro2, which calls into proc_macro.
Very importantly, this doesn't work, in the strictest sense. It's attempted to check whether it works, and the corresponding panic is silenced, but it has a side-effect.
That side-effect happens to be "defining" Bridge::with for the main executable, in the dynamic linker, with the body using the specific static that is linked into the main executable.
And in some circumstances (glibc before 2.27, or some other yet-to-be-determined part of the toolchain being before a certain version, RTLD_DEEPBIND not being enabled, and using the dylib or proc-macro crate type, not cdylib), that "definition" is used to resolve an intra-translation-unit call in the .so that gets dynamically loaded.

This is what i tried to say in my previous post :)

vlad20012 · 2019-06-05T09:35:07Z

Why proc-macro is dylib instead of cdylib if it (should?) export C ABI symbols only?

eddyb · 2019-06-05T10:07:58Z

Oh dear, it looks like it's #54592 (comment).

eddyb · 2019-06-05T10:09:15Z

@vlad20012 It should be more like cdylib, but with Rust metadata (like dylib).
However, that doesn't matter, both should work.

eddyb · 2019-06-05T10:25:51Z

This is what i tried to say in my previous post :)

@edwin0cheng Yeah, I realize now, and I'm sorry it took me this much, but there were too many levels of confusion, before I arrived at the reduction with one static and one getter fn.

eddyb · 2019-06-05T10:33:13Z

Opened #61539 - not sure we want to close the issue now, we might want to wait for some sort of fix.

Btw, because this is caused by #54592, you can also pass -Z plt=yes as a rustc flag when building the binary that does the dynamic loading, and it will also fix the issue (confirmed by testing).

hawkinsw · 2019-08-19T19:35:58Z

I can confirm that RTLD_DEEPBIND indeed resolved the issue for me, thanks @edwin0cheng tada

Maybe the whole problem wasn't noticed in the compiler itself because the compiler and the procedural macros are compiled by different compilers after all? I mean that the final stage of the compiler is compiled with the previous stage compiler, and the macros is compiled with the final stage compiler and then is linked into it

In my example, the panic occurs only when the same compiler is used for the procedural macro and for the linking code compilation (and without RTLD_DEEPBIND). Maybe if we use the same compiler to compile the compiler itself and procedural macro, and then will use this new compiler and procedural macro together, we will see the same behavior?

Amazingly enough, I, too, am seeing a similar error message! Can you show/tell how you applied the RTLD_DEEPBIND flag to the rustc call so that I can see if it fixes my issue, too? If it does, I will submit more details about my use case to this bug so that it might be possible to see if the fix(es) proposed here will address both problems!

Thanks for all the work that everyone is doing to dig into this problem!
Will

3920: Implement expand_task and list_macros in proc_macro_srv r=matklad a=edwin0cheng This PR finish up the remain `proc_macro_srv` implementation : 1. Added dylib loading code for proc-macro crate dylib. Note that we have to add some special flags for unix loading because of a bug in old version of glibc, see fedochet/rust-proc-macro-panic-inside-panic-expample#1 and rust-lang/rust#60593 for details. 2. Added tests for proc-macro expansion: We use a trick here by adding `serde_derive` to dev-dependencies and calling `cargo-metadata` for searching its dylib path, and expand it in our tests. [EDIT] Note that this PR **DO NOT** implement the final glue code with rust-analzyer and proc-macro-srv yet. Co-authored-by: Edwin Cheng <edwin0cheng@gmail.com>

eddyb · 2022-07-29T22:10:18Z

Not sure yet if relevant, but: turns out my expectation of proc macros being "most cdylib" is wrong because (more details here: #99909 (comment)) of this bail-out:

rust/compiler/rustc_codegen_ssa/src/back/linker.rs

Lines 651 to 661 in 9067d52

 // We manually create a list of exported symbols to ensure we don't expose any more. 

 // The object files have far more public symbols than we actually want to export, 

 // so we hide them all here. 

 if !self.sess.target.limit_rdylib_exports { 

 return; 

 } 

 if crate_type == CrateType::ProcMacro { 

 return; 

 }

AFAICT that's what's causing proc macro dylibs to have so many dynamic exports (instead of exactly one, the proc_macro_decls or w/e it's called) - and if all it takes for issues like this to occur is dynamic linking to unify one of those symbols, that might be what's been causing all of these issues for years and we just didn't look in the right place...

eddyb · 2022-08-02T07:25:33Z

@fedochet @edwin0cheng @hawkinsw Checking in now that this landed:

Limit symbols exported from proc macros #99944

Can any of you confirm that:

nightly-2022-08-01 (before that PR) can reproduce the issue
nightly-2022-08-02 (after that PR) does not reproduce anymore

We can likely close this issue once confirmed. Thanks in advance!

edwin0cheng · 2022-08-08T07:48:05Z

Thanks @eddyb for your investigation and fixes for this issue !

And since the proc-macro bridge in rust was changed a lot, the original example code from @fedochet is obsoleted.
But I spent some times to recreate a new one which based on Docker to use an old libc version.

And it confirmed your fixes works !

I think we could close this issue.

eddyb · 2022-08-08T23:31:20Z

@edwin0cheng Well, you should thank @bjorn3 ;) - I'm happy it's over though!

fedochet mentioned this issue May 6, 2019

proc_macro: stop using LEB128 for RPC. #59820

Merged

edwin0cheng mentioned this issue May 26, 2019

Some notes about the panic fedochet/rust-proc-macro-panic-inside-panic-expample#1

Open

eddyb mentioned this issue Jun 5, 2019

1.30 -> 1.31 dylib late-binding regression with GNU binutils 2.28 or older. #61539

Closed

jonas-schievink added A-macros Area: All kinds of macros (custom derive, macro_rules!, proc macros, ..) C-bug Category: This is a bug. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Aug 6, 2019

hawkinsw mentioned this issue Aug 22, 2019

use of proc_macro[2]::TokenStream in a vector in TLS causes rustc panic #63804

Open

pnkfelix mentioned this issue Nov 29, 2019

For dylib crates, warn about GNU ld <=2.28 #66839

Closed

edwin0cheng mentioned this issue Apr 9, 2020

Implement expand_task and list_macros in proc_macro_srv rust-lang/rust-analyzer#3920

Merged

bjorn3 closed this as completed Aug 8, 2022

Panic inside panic when procedural macro is called with proc_macro::bridge::client #60593

Panic inside panic when procedural macro is called with proc_macro::bridge::client #60593

Comments

fedochet commented May 6, 2019

eddyb commented May 6, 2019

fedochet commented May 7, 2019

eddyb commented May 7, 2019

fedochet commented May 7, 2019

eddyb commented May 13, 2019

fedochet commented May 13, 2019

eddyb commented May 14, 2019

edwin0cheng commented May 28, 2019 • edited Loading

eddyb commented May 28, 2019

edwin0cheng commented May 30, 2019 • edited Loading

eddyb commented May 30, 2019

edwin0cheng commented May 31, 2019 • edited Loading

edwin0cheng commented May 31, 2019

eddyb commented Jun 2, 2019

edwin0cheng commented Jun 2, 2019

fedochet commented Jun 2, 2019 • edited Loading

eddyb commented Jun 2, 2019

edwin0cheng commented Jun 2, 2019 • edited Loading

eddyb commented Jun 2, 2019

eddyb commented Jun 3, 2019 • edited Loading

fedochet commented Jun 3, 2019

eddyb commented Jun 3, 2019 • edited Loading

fedochet commented Jun 3, 2019 • edited Loading

eddyb commented Jun 3, 2019

eddyb commented Jun 3, 2019 • edited Loading

fedochet commented Jun 4, 2019

eddyb commented Jun 5, 2019 • edited Loading

eddyb commented Jun 5, 2019 • edited Loading

edwin0cheng commented Jun 5, 2019 • edited Loading

vlad20012 commented Jun 5, 2019

eddyb commented Jun 5, 2019

eddyb commented Jun 5, 2019

eddyb commented Jun 5, 2019

eddyb commented Jun 5, 2019

hawkinsw commented Aug 19, 2019

eddyb commented Jul 29, 2022

eddyb commented Aug 2, 2022

edwin0cheng commented Aug 8, 2022 • edited Loading

eddyb commented Aug 8, 2022

edwin0cheng commented May 28, 2019 •

edited

Loading

edwin0cheng commented May 30, 2019 •

edited

Loading

edwin0cheng commented May 31, 2019 •

edited

Loading

fedochet commented Jun 2, 2019 •

edited

Loading

edwin0cheng commented Jun 2, 2019 •

edited

Loading

eddyb commented Jun 3, 2019 •

edited

Loading

eddyb commented Jun 3, 2019 •

edited

Loading

fedochet commented Jun 3, 2019 •

edited

Loading

eddyb commented Jun 3, 2019 •

edited

Loading

eddyb commented Jun 5, 2019 •

edited

Loading

eddyb commented Jun 5, 2019 •

edited

Loading

edwin0cheng commented Jun 5, 2019 •

edited

Loading

edwin0cheng commented Aug 8, 2022 •

edited

Loading