Skip to content

Commit

Permalink
Use non-blocking log writer (#6470)
Browse files Browse the repository at this point in the history
This will utilize a separate thread to write out the spans and events
without while letting the main computation to proceed with its business.
Additionally, we are buffering the output by lines, thus reducing the
frequency of syscalls that can occur when the subscriber is writing out
parts of the message.

This should mitigate concerns of enabling debug logging as its impact on
performance should now be minimal (putting an event structure onto a
MPSC queue.) There are still costs associated with logging everything
however. Most notably formatting and construction of the
`tracing_core::ValueSet`s still occur on the caller side, so if
constructing those is expensive, the logging might remain expensive.
An example of code sketchy like that (although silly) could be:

```
debug!(message = { std::time::sleep(Duration::from_secs(1)); "hello" })
```

or for a less silly example:

```
debug!("{}", my_vector.iter().map(|...| {
  do_expensive_stuff()
}).collect::<String>())
```

These should be considered a bug (alas one that `tracing` does not have
any tooling to detect, sadly.)

I opted adding a new crate dedicated to observability utilities. From my
experience using things like prometheus will eventually result in a
variety of utilities being written, so this crate eventually would
likely expand in scope...

Fixes #6072 (though I haven't made any actual measurements as to what the improvement really is)
  • Loading branch information
nagisa authored Mar 25, 2022
1 parent 0a94f85 commit 6351eb5
Show file tree
Hide file tree
Showing 19 changed files with 283 additions and 159 deletions.
119 changes: 78 additions & 41 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ members = [
"core/primitives-core",
"core/store",
"core/metrics",
"core/o11y",
"runtime/near-vm-logic",
"runtime/near-vm-runner",
"runtime/near-vm-runner/fuzz",
Expand Down
14 changes: 14 additions & 0 deletions core/o11y/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
[package]
name = "near-o11y"
version = "0.0.0"
authors = ["Near Inc <hello@nearprotocol.com>"]
description = "Observability helpers for the near codebase"
edition = "2021"
publish = false
readme = "README.md"
rust-version = "1.56.0"

[dependencies]
tracing = { version = "0.1.13", features = ["std"] }
tracing-subscriber = { version = "0.3.9", features = ["fmt", "env-filter", "std"] }
tracing-appender = "0.2.2"
9 changes: 9 additions & 0 deletions core/o11y/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
Observability (o11y) helpers for the NEAR codebase.

This crate contains all sorts of utilities to enable a more convenient observability implementation
in the NEAR codebase. Of particular interest to most will be standardized tracing subscriber setup
available via the [`default_subscriber`] function.

Among other things you should expect to find here eventually are utilities to work with prometheus
metrics (wrappers for additional metric types, a server publishing these metrics, etc.) as well as
NEAR-specific event and span subscriber implementations, etc.
Loading

0 comments on commit 6351eb5

Please sign in to comment.