-
-
Notifications
You must be signed in to change notification settings - Fork 258
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Pex files do not allow for OpenTelemetry auto-instrumentation #2065
Comments
@jsirois helpfully showed me how I could make OpenTelemetry work within a Pex, with the auto-instrumentation being run as the Pex entrypoint: https://gist.github.com/Peder2911/4842c21c131c795d24e4f76883e9b315 This is fine, though I would really like to make this process external to Pex for the above-mentioned reasons (keeping telemetry out of the way of developers). Also, I could not get this to work in my Pants repo, but that might just be my lack of configuration skills showing: https://github.com/Peder2911/otel_pants_reprex |
This is how Opentelemetry monkey-patches to add instrumentation to libraries: |
The issue there is fixed by: diff --git a/apps/dockerfiles/app.dockerfile b/apps/dockerfiles/app.dockerfile
index 5b47ef0..7138bdb 100644
--- a/apps/dockerfiles/app.dockerfile
+++ b/apps/dockerfiles/app.dockerfile
@@ -3,4 +3,4 @@ FROM python:3.9
ARG PEX_FILE
COPY $PEX_FILE /app.pex
RUN PEX_TOOLS=1 /app.pex venv --non-hermetic-scripts --bin-path prepend /app/venv
-ENTRYPOINT ./app.pex --metrics_exporter console --traces_exporter console uvicorn asciinator.app:app --host 0.0.0.0
+ENTRYPOINT /app/venv/pex --metrics_exporter console --traces_exporter console uvicorn asciinator.app:app --host 0.0.0.0 In other words, you created a non-hermetic venv successfully, but then missed using it as the entrypoint - you mistakenly used the original PEX. |
There is no need to make the default entrypoint of the PEX be As far as the need to include opentelemetry in the PEX, I'm not sure that can be eliminated. I'll dig a bit more and report back. |
So opentelemetry deps definitely need to be included in the PEX. But, with a quick hack session to extend the
That yields a PEX that, run normally, just functions like a flask app:
But, when run with the alternate
|
This provides the same option when building a PEX as the `--non-hermetic-scripts` does when creating a venv from a PEX using `pex-tools`. In addition, both means of creating a non-hermetic venv are extended to the venv `pex` script itself. Closes pex-tool#2065
This provides the same option when building a PEX as the `--non-hermetic-scripts` does when creating a venv from a PEX using `pex-tools`. In addition, both means of creating a non-hermetic venv are extended to the venv `pex` script itself. Closes #2065
Alright @Peder2911, with You build an opentelemetry PEX one time and include this in a base image:
N.B.: The opentelemetry PEX should include all Then, each app that wants to be instrumentable looks like so:
In this case the critical options are, again With that setup, you instrument a random app like so:
In summary:
|
Actually, by including a Creating an pex \
opentelemetry-distro \
opentelemetry-instrumentation-flask \
pex \
--no-strip-pex-env \
--venv \
--non-hermetic-venv-scripts \
-c pex-tools \
-o opentelemetry.pex Allows you to do this to any PEX with no restrictions on how the PEX was built and not knowing the PEX entrypoint: RUN \
PEX_PATH=opentelemetry.pex \
./opentelemetry.pex \
./app.pex \
venv \
--non-hermetic-scripts \
app/venv And that venv can then be run auto-instrumented via: ENTRYPOINT \
app/venv/bin/opentelemetry-instrument \
--traces_exporter console \
--metrics_exporter console \
./app/venv/pex |
This is a really elegant solution @jsirois, will try this out in my repo right away! |
Beautiful! It works! |
We are using pantsbuild to set up a developer platform with an emphasis on a nice, frictionless developer experience. A part of this experience is codeless auto-instrumentation such as is provided by Opentelemetry with auto instrumentation. Specifically, we are looking to auto-instrument using the Kubernetes Operator offered by OpenTelemetry.
Auto-instrumentation with OpenTelemetry works by monkey-patching libraries, creating passthrough functions that wrap existing functions with logging statements. The way this works is kind of obscure to me, and is something I have to figure out a bit more before I can give constructive input into how to support this.
Auto-instrumentation of non-pexified applications can be as easy as running (for example from a clean Python3.9 docker container):
This will spin up a Flask server that also emits telemetry. This separation of code and logging is super useful for maintaining a clean developer experience.
This does not work out-of-the-box with Pex, however. Ideally, one would be able to run the following to achieve the same thing:
This does, as expected, not allow OpenTelemetry to inject any instrumentation ,and will not result in any telemetry being emitted. It would be pretty awesome if it did though! I would appreciate any thoughts and ideas for how to implement this.
The text was updated successfully, but these errors were encountered: