[Proposal] opentelemetry-go-auto-instrumentation

[D-D-H]

Description

The opentelemetry-go-auto-instrumentation project is an auto-instrumentation solution designed for Go applications. It empowers users to harness the capabilities of OpenTelemetry for enhanced observability without any manual modifications.
Like the opentelemetry-java-instrumentation project, this solution automatically modifies code, the difference is that this all happens during the build process.
The current implementation reuses the existing instrumentation for Go packages and depends on the package dave/dst to rewrite Go source code.
The side effect of this solution is similar to the impact one would expect from manual code modifications:

• Increased the final binary size.
• Introduced some performance overhead.

Benefits to the OpenTelemetry community

This project significantly lowers the barrier for Go applications to adopt OpenTelemetry.
While there is an existing auto-instrumentation solution based on eBPF, it comes with certain limitations.
Auto-instrumentation based on code rewriting can achieve the same effect as manual instrumentation in most scenarios and is easier to use in production.

Reasons for New Project

Drawing inspiration from the Java language, users generally prefer non-intrusive solutions (those that don't require manual code modifications). Therefore, we believe that for Go applications, this approach is likely to gain widespread acceptance among users. Making it a project of OpenTelemetry, not only ensures better maintenance but also extends the benefits to a broader user base.

Repository of Our Prototype

https://github.com/alibaba/opentelemetry-go-auto-instrumentation

Existing usage

This project is under development and has some simple demos.

Maintenance

The original contributors to this repository will continue to be involved in the project.
Our current roadmap is as follows：

1. Support all packages listed in open-telemetry/opentelemetry-go-contrib.
2. Instrument user code according to user-defined configuration.
3. Instrument Go runtime if it is necessary.

Licenses

Apache License 2.0

Trademarks

No Trademarks

Other notes

No response

[edeNFed]

In addition to the eBPF solution there is also a code generation solution called instrgen:
https://github.com/open-telemetry/opentelemetry-go-contrib/tree/main/instrgen

cc @pdelewski

[jiekun]

Before diving deeper, may I kindly ask if this would be similar in implementation to:

• https://github.com/apache/skywalking-go
• https://github.com/open-telemetry/opentelemetry-go-contrib/tree/main/instrgen

Edit: I have great respect for every project and idea, but based on previous observations, their biggest challenge often lies in the lack of a sufficient number of contributors to sustain active development. However, regardless of that, I am extremely grateful for the approach of auto-instrumentation in Go applications and I hope it can be sustained.

[pdelewski]

The idea described by this proposal is generally the same as both projects mentioned above by @jiekun. We should rather focus on what we have so far, build larger community around it and improve. I would be very happy to see more people contributing.

[D-D-H]

Thank you all for the comments.

Sorry, I didn't notice instrgen before. I'll take some time to learn about it and see how we can improve it together.

As @pdelewski said, @jiekun The principle of rewriting code of these projects you mentioned should be similar.

[jmacd]

Thank you @D-D-H. There has been a Go-auto-instrumentation started, with two approaches led by @edeNFed and @pdelewski. I would like to see you join the Go-auto-instrumentation group, which still has a meeting on the OpenTelemetry calendar Tuesdays, 9:30am PST to discuss this proposal. If this group does not have critical mass, we recommend moving this sort of topic into the Go SIG meeting on Thursdays at 10am PST.

[D-D-H]

@jmacd
Sorry for the late response and thank you for the invitation.
We may not be able to participate in this meeting. How about the next meeting?

[pdelewski]

@D-D-H Feel free to join any time you want.

[ralf0131]

Hi, @jmacd @pdelewski @edeNFed I am an engineer working on observability in Alibaba Cloud, I have summarized the main difference between our approach and instrgen and the eBPF solution to the best of our knowledge and please correct me if I am wrong.

Comparison with Instrgen

InstrGen leverages Golang's toolexec capability to inject instrumentation code at compile time. With minor changes to the code, users could get auto instrumentation for various libraries. Additionally, it offers crucial support for asynchronous context propagation, a feature highly valued by users. While InstrGen offers several benefits, some aspects may require further evaluation for more user adoption:

• Manual Code Modification: Currently, users need to manually add a line of code to their main function. While it seems minor, according to our experience, some users might prefer a solution that doesn't directly modify their codebase.
• When users upgrade libraries beyond versions compatible with InstrGen, the injected instrumentation may revert the library to a previously supported version. This behavior deviates from the expected upgrade experience.
• Dependency conflict: InstrGen's instrumentation code might introduce dependencies that could conflict with existing user-side dependencies.

Similar to InstrGen, our approach leverages compiler injection to insert instrumentation code. This approach offers several key advantages for users:

• No user-side code modifications are required for the instrumentation to work.
• Enhanced context propagation: it does not require user to ensure the context is correctly passed, it also support context propagation in asynchronous scenarios like goroutines.
• If users have previously added custom spans using the OTel-sdk within their code, our approach integrates these custom spans into the generated trace data.
• Enhanced Backward Compatibility: Unlike OTel-go-sdk which only supports the latest two Go releases, our instrumentation offers broader backward compatibility with older versions of instrumented libraries. This eliminates the need for users to manage specific library versions based on OTel-go-sdk limitations.

Comparison with the opentelemetry-go-instrumentaiton (eBPF solution)

The opentelemetry-go-instrumentation project leverages eBPF uprobes for non-intrusive instrumentation of Go applications. Currently supported library includes, net/http, grpc, kafka, SQL and etc. The benefits of this approach are:

• Automatic detection of go process and auto instrumentation.
• No code modifications are required.
• Support wide range of go versions.
• Support context propagation.

Actually we have tried the eBPF approach for a while, the considerations that we did not adopted this approach:

• Potential Performance Impact: Uprobes can introduce context switching between user and kernel modes, which may have additional performance implications for applications.
• Kernel and eBPF Constraints: Compatibility relies on the user's kernel version and due to the limitation of eBPF programs, extending support for additional libraries can be a relatively complicated process.
• Limited Context Propagation: Due to limitations in the eBPF instruction set, the maximum number of HTTP headers that can be propagated is currently limited (e.g., 8 for net/http). This may not be sufficient for all production use cases. The solution also require the user to ensure the context is correctly passed. However, in many cases, users often could not guarantee it. This makes the trace context not correctly propagated, especially in the asynchronised scenarios.

Please feel free to comment if you have any questions. We would like to have more discussion with the community in the upcoming SIG meeting. However, the meeting time is not quite friendly with us. I wonder is there any Asia-pacific friendly time for the meeting?

[pdelewski]

@ralf0131 From instrgen perspective as it uses basically the same technology as your approach we can combine them to have best of both. Regarding advantages that you mentioned:

• I don't think user source code modification is needed. This was initial approach, but we can eliminate that.
• regarding enhanced context propagation, I'm not sure how your approach is different from insrgen one and would like to understand it better.
  Regarding last two advantages as I said, there is no limitations from instrgen to be on par.

Currently the most important problem we are struggling is to have more people contributing to compile time instrumentation. CC @MrAlias @pellared

[ralf0131]

@pdelewski Please see my comments in line.

@ralf0131 From instrgen perspective as it uses basically the same technology as your approach we can combine them to have best of both.

I agree that the two approaches share the basic idea however the implementation may varies. I think maybe we can discuss about how to combine them. However due to the difference between the two approaches I am not quite sure how to do that.

Regarding advantages that you mentioned:

• I don't think user source code modification is needed. This was initial approach, but we can eliminate that.

That will be good. How would you like to do that?

• regarding enhanced context propagation, I'm not sure how your approach is different from insrgen one and would like to understand it better.

Based on my understanding, assuming user has a function called foo(), InstrGen will modified it to foo(ctx Context). Our approach does not modify the foo() function to add the Context as a parameter. Instead, it adds a dedicated field within the goroutine's definition, to store the span context. When a new span is created, the span context will be stored in the goroutine's field. Subsequent retrievals of the span context from the goroutine directly access the previously stored value. To support the asynchronous propagation, during goroutine creation, a child-context can be passed along, enabling context propagation even when user does not pass the context as parameter. We will further consider baggage propagation as well in the future. The idea here is like ThreadLocal in Java.

There is another difference regarding how the instrumented codes are injected. InstrGen analyze code from AutotelEntryPoint and build the call graph, and inject instrumentation code into functions bodies.
Our approach employs a rule base policy. It customize with toolexec and analyze all the dependencies. For the dependencies that match the rule, the instrumentation code will be injected.

Regarding last two advantages as I said, there is no limitations from instrgen to be on par.

Could you elaborate more on how InstrGen will do that?

Currently the most important problem we are struggling is to have more people contributing to compile time instrumentation. CC @MrAlias @pellared

I agree with that. :)

[pdelewski]

@ralf0131 Please see my comment below.

Based on my understanding, assuming user has a function called foo(), InstrGen will modified it to foo(ctx Context). Our approach does not modify the foo() function to add the Context as a parameter. Instead, it adds a dedicated field within the goroutine's definition, to store the span context. When a new span is created, the span context will be stored in the goroutine's field. Subsequent retrievals of the span context from the goroutine directly access the previously stored value. To support the asynchronous propagation, during goroutine creation, a child-context can be passed along, enabling context propagation even when user does not pass the context as parameter. We will further consider baggage propagation as well in the future. The idea here is like ThreadLocal in Java.

There is another difference regarding how the instrumented codes are injected. InstrGen analyze code from AutotelEntryPoint and build the call graph, and inject instrumentation code into functions bodies.
Our approach employs a rule base policy. It customize with toolexec and analyze all the dependencies. For the dependencies that match the rule, the instrumentation code will be injected.

Could you elaborate more on how InstrGen will do that?

Your understanding is based on what we have so far on the main branch, however there is PR open-telemetry/opentelemetry-go-contrib#4058 (opened almost year ago) which is more or less what you described above, so both tools follows the same techniques. Of course there still might be some implementation differences, but they are rather small from high level view.

[ralf0131]

Could you elaborate more on how InstrGen will do that?

Your understanding is based on what we have so far on the main branch, however there is PR open-telemetry/opentelemetry-go-contrib#4058 (opened almost year ago) which is more or less what you described above, so both tools follows the same techniques. Of course there still might be some implementation differences, but they are rather small from high level view.

Hi @pdelewski ,

You're right. My apologies, I hadn't noticed your pull request before and my understanding are based solely on the main branch.

I've taken a look at your implementation, and I agree that from the high level view they share the same idea. We're actually planning to open-source the latest version of our code sometime in June. This would allow us for a more in-depth discussion about combining the two approaches once we can see each other's work in detail. What do you think?

[pdelewski]

Hi @ralf0131 ,

That's great idea. Also from timeframe perspective, June sound good to me.

[jiekun]

I saw a great idea and comments from maintainers here. As we know, providing auto-instrumentation for Go programs is difficult, but that's not the reason which cause the previous implementation being less active.

I've talked to some developers before, and they haven't heard about instrgen. I assume they will also be not aware of the potential implementation discussed here. So, there are a couple things we need to consider:

• How can we incubate this (both instrgen and potentially a new implementation)? How can we increase attract more users? (External talks? Blog post?)
• How can we advocate for more developers and maintainers to join these projects?
• ...

It would be great to see new projects/impls, and please also consider how we could keep them active for a long period of time.

[ralf0131]

I've talked to some developers before, and they haven't heard about instrgen. I assume they will also be not aware of the potential implementation discussed here. So, there are a couple things we need to consider:

I saw a great idea and comments from maintainers here. As we know, providing auto-instrumentation for Go programs is difficult, but that's not the reason which cause the previous implementation being less active.

I've talked to some developers before, and they haven't heard about instrgen. I assume they will also be not aware of the potential implementation discussed here. So, there are a couple things we need to consider:

• How can we incubate this (both instrgen and potentially a new implementation)? How can we increase attract more users? (External talks? Blog post?)

That's right, I think we will try to add some documentation and articles to introduce the compile time instrumentation. We also submitted proposal to the KubeCon China 2024 and hopefully we can be there to present :).

[pdelewski]

Let me share my thoughts.

It seems that most people focus on ebpf right now, no matter that it's just harder from development perspective and has tradeoffs described above. Having said that, it has one advantage that might be important for some group of people, e.g no need for recompilation or access to source code.

Another thing is that it's unfortunate that instrgen landed in opentelemetry-go-contrib instead of opentelemetry-go-instrumentation so people interested in go instrumentation usually go there and don't know about instrgen existence.

I also tried to advertise it during KubeCon 2023 in Chicago, however that's not enough. Blogposts, articles and more people involved might help change the situation.

[ralf0131]

Let me share my thoughts.

It seems that most people focus on ebpf right now, no matter that it's just harder from development perspective and has tradeoffs described above. Having said that, it has one advantage that might be important for some group of people, e.g no need for recompilation or access to source code.

Another thing is that it's unfortunate that instrgen landed in opentelemetry-go-contrib instead of opentelemetry-go-instrumentation so people interested in go instrumentation usually go there and don't know about instrgen existence.

Just be curious, may I ask why instrgen was landed in opentelemetry-go-contrib? Is it possible to move it to somewhere under opentelemetry-go-instrumentation ? I do think this a key reason people might be difficult to know about it. At least in the documentation, I did not find any official reference to instgen.

I also tried to advertise it during KubeCon 2023 in Chicago, however that's not enough. Blogposts, articles and more people involved might help change the situation.

[edeNFed]

I don't think moving instrgen to opentelemetry-go-instrumentation is a good idea.
From a user perspective, I expect all the repositories named opentelemetry-XXX-instrumentation will be an agent-like thing that does instrumentation in runtime (this is correct for Java, Python, .NET, JS).
Compile time instrumentation falls under the tool category and should therefore be in a different repository.

[ralf0131]

I don't think moving instrgen to opentelemetry-go-instrumentation is a good idea. From a user perspective, I expect all the repositories named opentelemetry-XXX-instrumentation will be an agent-like thing that does instrumentation in runtime (this is correct for Java, Python, .NET, JS). Compile time instrumentation falls under the tool category and should therefore be in a different repository.

@edeNFed Thanks for the clarification. Perhaps we should at least let users know, in the documentation, there are two approaches of instrumentation, one is the eBPF based auto instrumentation and the other is the compile time instrumentation?
My suggestion will be:

Go
\--GettingStarted
\--Instrumentation
\--Automatic
    \-- runtime instrumentation (eBPF based)
    \-- compile time instrumentation

Looking at the documentation of Java, there is also a Spring Boot page under automatic folder, which is not a fully runtime instrumentation. To my understanding, if a Java application is compiled into a native image with GraalVM, this process is essentially a compile time instrumentation.

My second question for the eBPF-based instrumentation is, eBPF could offer a approach to instrumentation that beyond any specific programming language, why it has not been applied to other languages besides Go?

[trask]

From a user perspective, I expect all the repositories named opentelemetry-XXX-instrumentation will be an agent-like thing that does instrumentation in runtime (this is correct for Java, Python, .NET, JS).

just fyi opentelemetry-java-instrumentation includes both agent (runtime instrumentation) and library (build-time instrumentation)

[jiekun]

I expect all the repositories named opentelemetry-XXX-instrumentation will be an agent-like thing that does instrumentation in runtime (this is correct for Java, Python, .NET, JS).

From my perspective as a user, I don't think so.

The opentelemetry-xxx-instrumentation repository should provide automated instrumentation solutions, whether it's eBPF, Java agent, compile-time instrumentation, or something else. It different from manual instrumentation, which should be placed under the opentelemetry-xxx(-contrib) repository.

Developers do not need to care about how Java's auto-instrumentation works. All they need to know is that they don't need to modify codes.

In short, I expect the solutions provided by the opentelemetry-xxx-instrumentation repository to not require code changes, or at most, only minor code modifications.

[edeNFed]

The only thing shared between the eBPF instrumentation and the compiled time instrumentation is that they are both targeting Go applications.

Everything else is different, the programming language the instrumentation are written in, the contributors working on the project, tests, CI, etc.

I agree with @ralf0131 if the goal is to get more visibility into compile time instrumentation, making it more visible in the documentation is preferred in my opinion instead of mixing two unrelated projects into single repository.

[trask]

The opentelemetry-xxx-instrumentation repository should provide automated instrumentation solutions, whether it's eBPF, Java agent, compile-time instrumentation, or something else. It different from manual instrumentation, which should be placed under the opentelemetry-xxx(-contrib) repository.

(sort of a side conversation, but just wanted to mention that it's not quite this clear of a distinction for the Java repos at least, where the primary differentiator is that opentelemetry-java-instrumentation modules are maintained by the repo maintainers while opentelemetry-java-contrib is a distributed ownership model where individual components are maintained by component owners)

[ralf0131]

Hi all,

Update: recently we have just released our first version of compile time instrumentation on the commercial side. Now we are working on the open source our solution. It is a bit late as expected but we are working on it.

[RomainMuller]

I'd like to note that @DataDog is up to extremely similar work in https://github.com/DataDog/orchestrion. This currently injects instrumentation for @DataDog's "properietary" SDK (https://github.com/DataDog/dd-trace-go)...

That said, the code rewriting is configuration-driven, and configuration targeting the OTel SDK instead could absolutely be made. We (at @DataDog) would most certainly welcome contributions in this direction.

[ralf0131]

HI All,

Update: we have almost finished the initial version the compile time instrumentation approach. We are heading towards the first release. We are planning to introduce the approach in the upcoming tag-observability meeting (Tuesday, Aug 13 · 18:00 – 19:00 (Time zone: America/Los_Angeles), Wednesday, Aug 14 · 9:00 – 10:00 (Time zone: UTC+8)). Anyone who is interested please join to discuss. The meeting information can be found here.

[danielgblanco]

Hi @ralf0131 any feedback from the tag-observability meeting you can share here?

[ralf0131]

@danielgblanco Not yet. Due to some technical issues, the meeting was not held as expected :(. We are working on a new meeting to discuss with it. We really need help on how to reach out to more people who is interested in the community. Any suggestions will be appreciated.

Hi @ralf0131 any feedback from the tag-observability meeting you can share here?

[damemi]

Notes from the meeting today:

• Main reasons for donating are to reach more developers and raise familiarity for this approach
  • Have found use cases with Alibaba Cloud customers, but the solutions are not specific to Alibaba users
• We expect to keep this project separate from the existing eBPF approach
  • There will be little (if any) involvement from the eBPF instrumentation maintainers
  • New project repo name should include "compile" or something similar without being too long/confusing
• Keeping separate means that the new project/SIG needs maintainers from 2+ companies
  • Currently have @ralf0131 and their team, along with @pdelewski
  • @ralf0131 mentioned that there are other potential companies interested and will reach out to them to add their support here
• The merger with instrgen will essentially be a migration to the new code base, which is further along
• New SIG/project will set up its own meeting time that is friendlier to their maintainers, who are mostly EMEA/APAC
  • Any necessary coordination between the two SIGs can be worked out as needed, but right now they are different enough with enough work to keep their own meetings
  • Possibly rename the existing SIG meeting to something like "Go eBPF Auto-Instrumentation" for clarity
• Need a TC liaison
  • Suggested @jpkrohling as they are the current liaison fro the eBPF approach
  • Also have questions on legal/IP/copyright transfer

AIs:

• @ralf0131 reach out to other company maintainers to leave a comment here showing interest in approver/maintainership
• @ralf0131 and @MrAlias Work on community issue for official proposal
• @jpkrohling would you be interested in being TC liaison for this project?
• @damemi post these notes and action items

Please let me know if I missed anything

[XSAM]

The merger with instrgen will essentially be a migration to the new code base, which is further along

I guess the new project would also have the functionality of instrgen, so it basically provides two modes for instrumentation:

1. Code generation only. The code it generated before the standard go build, and generated code would be part of the codebase (tracked by git).
2. Compile time code injection. By using a customized go compiler, the generated code does not reflect in the codebase; the binary itself contains instrumentation.

Is that correct? @damemi

[damemi]

@XSAM my understanding on the call was more that we agreed to deprecate instrgen in favor of the new project. @ralf0131 or @pdelewski can correct me

[pdelewski]

@XSAM As @damemi written, we decided to deprecate instrgen in favor of this project and add features that are currently unique for instrgen. Code generation only is one of them.

[jiekun]

@ralf0131 reach out to other company maintainers to leave a comment here showing interest in approver/maintainership

Thanks for all your efforts. This proposal looks promising. I'm looking forward to cooperating and providing more options of instrumenting Go applications.

[JaredTan95]

@ralf0131 reach out to other company maintainers to leave a comment here showing interest in approver/maintainership

I have only used the golang instrumentation with ebpf, but recently we have been facing some problems due to kernel requirements. I am not aware of the instgen. I am very happy to try and use Compile-Time Instrumentation in this proposal.

[pdelewski]

@ralf0131 reach out to other company maintainers to leave a comment here showing interest in approver/maintainership

I recently spoke with @dineshg13 about helping with this effort, so we may be able to fulfill the requirement to have approvers/maintainers from two or more companies.

[dineshg13]

Happy to help with the effort.

[RomainMuller]

I'm also happy to contribute; been working on https://github.com/DataDog/orchestrion which is very much in that space.

[MrAlias]

@open-telemetry/technical-committee this proposal looks ready to be accepted:

• This is proposed as a new SIG dedicated to the compile-time instrumentation solution for Go.
  • The seed SIG maintainer are: @dineshg13, @pdelewski, and @ralf0131
  • The approver roles are expected to be decided by these maintainers.
  • This SIG is independent of the existing Go and Go auto-instrumentation SIGs.
• The code will be hosted in its own repository.
  • This repository needs creation.
• The existing instrgen code will be deprecated in favor of the new repository.

[ralf0131]

@MrAlias I'd like to confirm that is there any legal process that needs to be done to migrate the repository from Alibaba to OTel?

[MrAlias]

@MrAlias I'd like to confirm that is there any legal process that needs to be done to migrate the repository from Alibaba to OTel?

@open-telemetry/governance-committee

[felixge]

The donation process for OpenTelemetry is described here: https://github.com/open-telemetry/community/blob/main/guides/contributor/donations.md

[reyang]

@open-telemetry/technical-committee this proposal looks ready to be accepted:

@MrAlias I'll bring this to the Wed. TC meeting and inform the GC as well.

[jpkrohling]

I'm happy to be the GC liaison for this.

[pellared]

Related issue in Go:

• proposal: cmd/go: compile-time instrumentation and -toolexec golang/go#69887

[y1yang0]

@open-telemetry/technical-committee this proposal looks ready to be accepted:

@MrAlias I'll bring this to the Wed. TC meeting and inform the GC as well.

Hi @reyang @MrAlias @jpkrohling , is there any update? Thanks.

[codefromthecrypt]

@D-D-H happy to help on the dev/x and story telling side of this. I have been excited about this project since learning about it at kubecon china. I want to help showcase Go+otel stories and this is a particularly interesting avenue. You have my support!

[D-D-H]

@codefromthecrypt Very glad to hear that you are interested in this project. Sharing more about this project would indeed be incredibly valuable. If you have any specific plans, please do let us know. We'd love to collaborate.

[codefromthecrypt]

If you have any specific plans, please do let us know. We'd love to collaborate.

I am assembling a kubecon submission for rejection... err I mean acceptance ;)

[D-D-H]

If you have any specific plans, please do let us know. We'd love to collaborate.

I am assembling a kubecon submission for rejection... err I mean acceptance ;)

We are also preparing the submission for KubeCon 2025 European conference.