add asyncio instrumentation

[bourbonkk]

Before opening a feature request against this repo, consider whether the feature should/could be implemented in the other OpenTelemetry client libraries. If so, please open an issue on opentelemetry-specification first.

Describe the solution you'd like
I want to add instrumentation to asyncio.
Has this been considered already?

Describe alternatives you've considered
Which alternative solutions or features have you considered?

https://github.com/aio-libs/aiomonitor/tree/main

I found a project for monitoring asyncio.
Maybe we could use that project and modify it for opentelemetry?
If it hasn't been considered, I'd like to try it.

[aabmass]

found a project for monitoring asyncio.
Maybe we could use that project and modify it for opentelemetry?
If it hasn't been considered, I'd like to try it

That sounds great I'd be interested to see this. Feel free to send a PR. We may be slow to review but I'd be happy to take a look

[nhtgl]

I was wondering how instrumenting asyncio aligns with ideas behind otel. Here are some of my thoughts.

Otel is not saying this explicitly, but leans towards instrumenting distributed systems. When you look at the instrumentations that already are in this repo they're typically* in or outs - http frameworks, database, cache, libs for sending requests (http, grpc, etc) somewhere else (that leave our system/microservice). This is to provide us with information that allow correlating things that happen in one system with things that are triggered by us in other system (or other way around), so we can understand better the chain of events.

Now, how does asyncio fit in this picture? Well, asyncio is not in or out when we think of our application. Surely, it can be treated as in or/and out thing, but in some isolation - not at application level. It fits more in "processing" category, because it sits in the middle.

Practical aspects would be:

• I can see that if this is merged, others would add instrumentation for trio, anyio, etc.
• It adds maintenance burden - please read this issue for more details Instrumentation acceptance policy and general guidelines for external instrumentation authors #1714
• Which metrics or spans names from the current specification fits this idea?

Issue request says

I want to add instrumentation to asyncio.

But can you tell us why? Maybe there's other way to achieve your goal without adding new instrumentation. I'm guessing that what you want to do is actually profile your application, probably don't care about RAM/CPU usage, but interested in execution time.

If that's the case then I can provide you with some help in this regard, because me and my team (ab)used OpenTelemetry to find out where we're getting stuck, where things are too slow, how we can optimize certain calls. This is in addition to in/out tracing.

To summarize, I'm not convinced we should add asyncio intrumentation to contrib library given the assumptions I made. You can of course create instrumentation repo for asyncio and publish on pypi for others to use.

*- jinja2 instrumentation doesn't fit this for me, at least how and what it's instrumented

[bourbonkk]

Hello, @macieyng

I appreciate your thoughtful feedback.

Indeed, I grappled with the dilemma of whether my contribution was apt. However, my commitment to adding instrumentation for asyncio stemmed from the following observations:
We employ asyncio extensively across a range of applications. While many established libraries utilize it, there remains a noticeable gap for async operations, particularly in areas like batch jobs developed outside of Otel-supported libraries. Debugging issues, such as logical failures and performance bottlenecks, have made observability a pressing concern. My awareness of this challenge heightened upon discovering projects like aiomonitor during my participation at PyCon 2023 in Korea.

Given OpenTelemetry's mission to bolster observability, I believe that it's pertinent to enhance its features, even if the need is identified by a single individual.

Naturally, I trust the discernment of our reviewers in this process. Thank you.

[nhtgl]

Okay. This gives us a better context and a use case.

If I understood correctly: you use pure Python for background jobs / tasks and want to monitor what happens. Is that right?

If so, then I get your point. However, I have a concern about how users will be using it. This should be addressed in the readme via providing at least one use case and how users shouldn't be using it. Beside that, I have no more comment atm.

I would like to learn more about how you utilize pure Python for background tasks - if that's what you do.

[bourbonkk]

I've added a few comments to the Asyncio-Instrumentation readme. Thank you for the valuable feedback. @macieyng

[aabmass]

I took a look at #1943, wanted to discuss here. I'm a bit concerned this instrumentation generate huge amounts of spans. These methods are pretty low-level and get called frequently. I still think these low level spans could be really cool, possibly as an opt-in with some like Jaeger flamegraphs.

Before reading the PR, I was expecting more "runtime metrics" e.g. UpDownCounter for number of outstanding tasks (queue size), total count of tasks created over time, etc.

[bourbonkk]

I fully share the concerns about producing huge amounts of spans. As macieyng mentioned, I recognize that this approach appears to diverge from otel's core principles. Objectively thinking, it might be judicious to either offer an option for selectively using instrumentation-asyncio or contemplate excluding this PR altogether. Leveraging runtime metrics as a representation seems like an apt choice (had I conceived this idea earlier, I might have taken that route in implementation).

If there are better directions or suggestions, I'd greatly appreciate your input.

[aabmass]

I don't think we need to throw away your PR, we could probably make the most fine-grained tracing off by default. As another datapoint, OTel JS has tracing of the fs module which can produce a ton of spans. I definitely see the value of fine grain spans for something like data pipeline orchestration.

[bourbonkk]

I'd like to make the following suggestions. If these ideas are deemed appropriate, I'll proceed with additional developments on this PR:

1. To mitigate concerns about producing huge amounts of spans, we can determine the creation of SPANs via an environment variable.

• Environment Variable: OTEL_INSTRUMENTATION_ASYNCIO_TRACE_ENABLED (of type bool, default is False)

2. Addition of runtime metrics related to asyncio(can change):

• Number of currently running coroutines
• Number of completed coroutines
• Average execution time of coroutines
• Number of tasks currently in queue
• Number of completed tasks
• Number of cancelled tasks
• Total number of tasks created
• etc.

3. Upon integrating opentelemetry-instrumentation-asyncio, metrics will be generated by default, and the decision to produce spans will be determined through the aforementioned configuration(OTEL_INSTRUMENTATION_ASYNCIO_TRACE_ENABLED).

If you have any insights or novel concepts, I would greatly appreciate your esteemed suggestions.

[aabmass]

Will discuss during the SIG meeting on Thursday. It's at 9AM PST–you're welcome to join but I realize the time doesn't work for everyone so I can bring it up as well.

[bourbonkk]

I understand. Even though I'm not good at English, I'll try to participate.
If I can't join, I'll refer to the public meeting notes.