tracing::Span::current() before global dispatcher init results in poisoned dispatcher state

[neoeinstein]

Bug Report

Invoking tracing::Span::current() from inside of a spawned tokio task prior to initializing the global dispatcher results in thread-local dispatchers being poisoned and returning NoSubscriber, resulting in dropped spans and log messages.

Version

tracing 0.1.37
tracing-attributes 0.1.23
tracing-core 0.1.30
tracing-error 0.2.0
tracing-futures 0.2.5
tracing-log 0.1.3
tracing-opentelemetry 0.18.0
tracing-serde 0.1.3
tracing-subscriber 0.3.16

Platform

Darwin [REDACTED] 21.6.0 Darwin Kernel Version 21.6.0: Thu Sep 29 20:12:57 PDT 2022; root:xnu-8020.240.7~1/RELEASE_X86_64 x86_64

Crates

tracing, tracing-core

Description

Discovered because the OpenTelemetry OTLP metrics exporter spins up a gRPC service which includes a tower::Buffer layer. That layer invokes tracing::Span::current() on a spawned thread as soon as the exporter is initialized. This initialization must happen before setting up the global dispatcher in order to add an appropriate layer into a Registry.

Repro code:

#[tokio::main]
async fn main() -> color_eyre::Result<()> {
    const POISON: bool = true;
    tokio::spawn(async move {
        let poisoning = format!("Poisoning {:?}", std::thread::current().id());
        if POISON {
            println!("{poisoning}");
            tracing::Span::current();
        } else {
            println!("NOT {poisoning}");
        }
    }).await?;

    tracing_subscriber::fmt::init();

    let handles = (0..10u16).map(|_| {
        tokio::spawn(async move {
            let s = tracing::info_span!("test");
            let _guard = s.enter();
            let span = tracing::Span::current();
            let thread = std::thread::current().id();
            let log = if span.is_none() {
                format!("{thread:?} NOT REAL")
            } else {
                format!("{thread:?} is fine")
            };
            println!("{log}");
        })
    }).collect::<Vec<_>>();
    for handle in handles {
        handle.await?;
    }

    return Ok(());
}

Output when POISON is false:

NOT Poisoning ThreadId(16)
ThreadId(13) is fine
ThreadId(17) is fine
ThreadId(16) is fine
ThreadId(13) is fine
ThreadId(15) is fine
ThreadId(12) is fine
ThreadId(13) is fine
ThreadId(17) is fine
ThreadId(14) is fine
ThreadId(11) is fine

Output when POISON is true:

Poisoning ThreadId(13)
ThreadId(13) NOT REAL
ThreadId(13) NOT REAL
ThreadId(15) NOT REAL
ThreadId(13) NOT REAL
ThreadId(13) NOT REAL
ThreadId(15) NOT REAL
ThreadId(15) NOT REAL
ThreadId(13) NOT REAL
ThreadId(15) NOT REAL
ThreadId(11) NOT REAL

[hawkw]

#2359 should fix this (along with a few related issues)