-
-
Notifications
You must be signed in to change notification settings - Fork 464
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Tracing extension doesn't work properly when used with async code #448
Comments
Thanks for the test, I have fixed it in |
Thank you for the fix, however, I don't think that would work though, as @D1plo1d has mentioned in #447 we need to wrap the future with |
I refer to the code implementation of |
Sorry, there still seems to be a problem. |
I still haven't found a solution. 😂 |
I am going to switch to |
@sunli829 I'm sorry to hear that. I'm not sure if this helps re: tracing but I think the key to their design is entering the span inside poll and dropping it at the end of poll. Entering span anywhere else will not work because it will wrap all futures that execute on that thread until it is dropped. The way they've built it every async thread is only inside the span for the period of a future's poll function - as soon as the poll function returns the drop handler exits the span. |
I think there is no problem with my last modification, but it just doesn't work. This is related to cross-threading. I ran into the same problem as described here, but the more troublesome thing is that I need a lot of Spans. |
My apologies if I'm missing something but looking at commit cf1fe6f it looks as though enter is being called outside of await. As I understand it this is not equivalent to instrument. As I understand it async_graphql currently handles spans like this: Thread 1:
How Instrument works:
|
Thank you for your effort, I think same problem would happen with opentelemtry. Pls check the opentelemtry async example https://github.com/open-telemetry/opentelemetry-rust/tree/main/examples/async they call |
I resumed the |
Released in |
Hi, thank you😁 gonna try it this week |
I've just updated! 😄 |
Hmm, do we still have a way to instrument websockets with a custom root span (eg. I'm using |
For a single request, it is easy to customize the parent span. But for subscriptions, asynchronous streaming is at a deeper level, maybe I should restore let span = tracing::info_span!("root");
Ok::<_, Infallible>(Response::from(
schema.execute(request).instrument(span).await,
)) |
I made a lot of changes, and now the Tracing extension is finally working great! |
Released in |
I think this issue has been resolved, if you find that it is still incorrect, please feel free to open it. |
Tracing still won't work with async graphql. The current implementation is not at all correct because it drops the Entered struct at the end of However, it is not as easy as simpling holding onto the return values from Instrument trait defines an Following is a simplified demonstration of how async-graphql is trying to handle tracing vs how it should handle it. use tracing::{span, Level};
use tracing_subscriber::{fmt::time::ChronoLocal, EnvFilter, FmtSubscriber};
use tokio::time::Duration;
use tracing::Instrument;
#[tokio::main]
async fn main() {
let filter = EnvFilter::from_default_env();
let subscriber = FmtSubscriber::builder()
// all spans/events with a level higher than TRACE (e.g, debug, info, warn, etc.)
// will be written to stdout.
.with_env_filter(filter)
// .json()
//.pretty()
.with_timer(ChronoLocal::rfc3339())
// completes the builder.
.finish();
tracing::subscriber::set_global_default(subscriber).expect("setting default subscriber failed");
let span = span!(Level::INFO, "my_tracing_demo");
let _s = span.enter();
// This is currently what async-graphql is doing right now
tokio::task::spawn(async move{
let span = span!(Level::INFO, "ONE");
let handle = span.enter();
example_async_fn_one().await
});
tokio::task::spawn(async move {
let span = span!(Level::INFO, "TWO");
let handle = span.enter();
example_async_fn_two().await
});
tokio::time::sleep(Duration::from_secs(30)).await;
/*
// This is what it should be doing
tokio::task::spawn(async move{
let span = span!(Level::INFO, "ONE");
example_async_fn_one().instrument(span).await
});
tokio::task::spawn(async move {
let span = span!(Level::INFO, "TWO");
example_async_fn_two().instrument(span).await
});
tokio::time::sleep(Duration::from_secs(30)).await;
*/
}
async fn example_async_fn_one(){
for i in 0..10 {
tracing::info!("hello from example one!");
tokio::time::sleep(Duration::from_secs(1)).await;
}
}
async fn example_async_fn_two(){
for i in 0..10 {
tracing::info!("hello from example two!");
tokio::time::sleep(Duration::from_secs(2)).await;
}
} If you run the example above, without the So, to fix this, I think we will have to instrument the resolution futures within the library. But I don't think we can get away with that without making tracing a core part of async-graphql(Not that I am opposed to it). Ultimately, it's impossible to implement it as an extension. |
In fact, my implementation is no different from |
Here is an example, it works very well. https://github.com/sunli829/async-graphql-tracing |
If I use fn resolve_start(&mut self, _ctx: &ExtensionContext<'_>, info: &ResolveInfo<'_>) {
let parent = match info.resolve_id.parent {
Some(parent_id) if parent_id > 0 => self.spans.get(&resolve_span_id(parent_id)),
_ => self.spans.get(&EXECUTE_CTX),
};
if let Some(parent) = parent {
let span = span!(
target: "async_graphql::graphql",
parent: parent,
Level::INFO,
"field",
id = %info.resolve_id.current,
path = %info.path_node,
parent_type = %info.parent_type,
return_type = %info.return_type,
);
self.enter_span(resolve_span_id(info.resolve_id.current), span);
}
} However, if I modify the hero resolver in your example by adding a call to impl QueryRoot {
async fn hero(
&self,
ctx: &Context<'_>,
#[graphql(
desc = "If omitted, returns the hero of the whole saga. If provided, returns the hero of that particular episode."
)]
episode: Episode,
) -> Character {
tracing::info!("I expect contextual information to be attached to this log");
tokio::time::sleep(std::time::Duration::from_secs(1)).await;
if episode == Episode::Empire {
Human(ctx.data_unchecked::<StarWars>().luke).into()
} else {
Droid(ctx.data_unchecked::<StarWars>().artoo).into()
}
} When you're creating a span and entering it, the fields attached to the span will be present in any subsequent events so long as the return value from Notice how Also, sorry about the delay. My laptop is a potato so compiling stuff takes a while lol. |
I understand the problem you encountered. Because of the lack of the current span, |
@sunli829 Super appreciate you having another look at this. I should have mentioned that my issues were also due to |
Thank you, I have actually considered the solution you put forward. It will definitely solve the problem, but it's just that the changes are too big to make me hesitate. 😂 |
Great, your already on it then! Just wanted to make sure that one hadn't been lost in all the back and forth. 😄 |
@sunli829 re: #447 I initially had concerns that every extension would need to wrap the execution future in some future they create (also my the way #447 is written made no sense if more then one As a bit of an amendment to #447 I realize now that if Thus there should not be any performance impact or extra wrapping futures for the other extensions that do not Edit: Words |
Sorry for delay, yeah unfortunately it doesn't work as expected. I see no way of putting tracing in the extensions. Have to move tracing inside the internal implementation with the feature flag(maybe?). I think its common practice to have tracing embedded into the code itself. Async code ref in tracing crate The way to make it work properly is with wrapping the futures with So for example we could wrap execute() pub async fn execute(&self, request: impl Into<Request>) -> Response {
let request = request.into();
let extensions = self.create_extensions(Default::default());
do_execute(..)
.await
.instrument(tracing::span!(
target: "async_graphql::graphql",
Level::INFO,
"execute"
))
.await
} or #[instument(name = "execute", level = "info", skip(self, request))]
pub async fn execute(&self, request: impl Into<Request>) -> Response {
.....
} That way we never have to care about passing or getting parent spans. So calling Regarding the async_hook I don't think lifetime references would be fixable. |
Good news, I have made a lot of progress! 🙂 |
@sunli829 I really appreciate the hard work you're putting into this project. I'm definitely going to help out when I get my new laptop. 👍 |
I have implemented a new extension mechanism on the |
Tested it out with my project. Works exactly as I expect it to, so far. I will keep testing it and will let you know if there are any issues. |
async fn execute(&self, ctx: &ExtensionContext<'_>, next: NextExtension<'_>) -> Response; Hells yes! ^_^ |
Expected Behavior
Should build correct spans tree
Actual Behavior
Inconsistent span trees, some spans are missing parent.
Don't think
span.with_subscriber(|(id, d)| d.enter(id))
works in multi threaded runtimeSteps to Reproduce the Problem
Specifications
The text was updated successfully, but these errors were encountered: