Introduce Layer Filtering

[davidbarsky]

Resolves #302.

Currently, spans and events can only be enabled/disable on a global subscriber basis. This PR adds per-layer filtering in tracing-subscriber, which means that layers can conditionally filter spans and events without impacting others layers. In practice, this enables layers to conditionally filter on metadata like source and levels and forward those spans and events to different outputs sinks. This also allows for the creation of sampling or rate-limiting filters.

With this PR, customers will be able to write something like:

let err_filter = FilterFn::new(|metadata, _| {
    metadata.level() == &Level::WARN || metadata.level() == &Level::ERROR
});
let info_filter = FilterFn::new(|metadata, _| metadata.level() <= &Level::INFO);

let err = fmt::Layer::builder()
    .with_writer(io::stderr)
    .finish()
let info = fmt::Layer::builder()
    .with_writer(io::stdout)
    .finish()

let subscriber = Registry::default()
    .with(info.with_filter(info_filter))
    .with(err.with_filter(err_filter));

tracing::subscriber::set_global_default(subscriber).expect("Unable to set global subscriber");

Still to do:

• Documentation
• Examples
• Implementing all of Layer's methods on Filtered.

[hawkw]

I think this also needs a register_callsite-esque hook (maybe call it filter_static? or something) so that filters can indicate that they never want to see something

[davidbarsky]

So that the interest level for each layer can be cached? That makes sense. I think that will be the primary interface for Filter.

[hawkw]

So that the interest level for each layer can be cached? That makes sense. I think that will be the primary interface for Filter.

so that the sum total interest can be cached

[hawkw]

if all filters will never be interested in a trace, it needs a cached never interest.
if any are sometimes interested, it probably should always get a cached sometimes
if all filters are always interested, it needs a cached always

[hawkw]

there may be some optimizations to quick-disable something when we know a given filter is the only "sometimes", but that seems more complex. i'm mostly concerned about all always and all never getting cached always and never interests, respectively...

[hawkw]

I think this needs to implement the enabled and register_callsite layer fns. I think we will also want to change the rules for how layers handle filters — I'd really like to get it so that if all the per-layer filters disable something, the call to enabled will return false.

[davidbarsky]

Yeah, I think this would require thinking a little about interest evaluation orders, which I have not done.

[davidbarsky]

@hawkw

As for:

if metadata.level().cmp(&self.level) == self.ordering {
    true
} else {
   false
}

...I've replaced that with a FilterFn struct in the style of ServiceFn. I think something like that should be exported by tracing-subscriber.

[davidbarsky]

@hawkw and I chatted about how to handle a few bugs that cropped up. Here's the approach that seemed to be the most composable (to me):

• Filtered needs to be a subscriber.
  • This allows filters to be applied to a single subscriber or multiple layers.
  • Callsite interest caching would work seamlessly.
• If a Layer has indicated that it is never interested in a span, it shouldn't ever see it again. This entails caching Layer interest on the Filtered struct or the private SpanData struct.

Some of this work was on this branch, if I recall correctly: https://github.com/tokio-rs/tracing/commits/eliza/filter-layers-extension. Additionally, instead of a Hashset, we were considering using a bitset to avoid hashes + enable really fast lookups. However, this does mean there is a maximum number of filters, but the numbers (512 or 1024) is so high, I don't expect any user of tracing would hit it. This approach would require an additional trait bound for Layer implementations, atop of the current S: Subscriber + for<'span> LookupSpan<'span>. In practice, the trait bound might be something like:

S: Subscriber + for<'span> LookupSpan<'span> + Filter

We also discussed an approach where a filter would be passed into every layer, meaning that layers would need to opt-in to being filtered. I don't think we opted for this approach.

[ufoscout]

Is there any chance that this is going to be merged soon?
Sorry for asking this, but we need exactly this feature for one of our products and I wonder if we can simply wait for a little to have it or we have to find a workaround :(

[hawkw]

Obsoleted by #1523.