Integrate spin_telemetry to support OTel

[calebschoepp]

Integrates spin_telemetry into the shim so that we get OTel support.

The workaround to prevent the panic is to point at versions of spin and spin triggers that don't use the tracing-log feature.

[calebschoepp]

spin_telemetry has a fmt_layer on the tracing subscriber that sends tracing events to STDERR. My testing shows that now that we have this fmt_layer if the user sets RUST_LOG=info they'll get additional logs. This is a good thing IMO.

@Mossaka do you know how environment variables are passed through to the shim in SpinKube? In my local testing I've been doing something like sudo ctr run ... --env RUST_LOG=info. In SpinKube is it as simple as setting an environment variable on the podspec of a SpinApp for it to be passed down to the shim? This isn't just relevant for setting something like RUST_LOG=info, but also determines how OTEL_EXPORTER_OTLP_ENDPOINT would be set.

[jsturtevant]

ENV variables are passed to shims via containerd. Set the ENV variable before starting containerd or configure the systemd service with the env's

[calebschoepp]

ENV variables are passed to shims via containerd. Set the ENV variable before starting containerd or configure the systemd service with the env's

Sorry I'm still not clear. Are you implying that in a K8s context the env variable needs to be set on the node? Is there a way to set it on the pod and have containerd pick it up?

[kate-goldenring]

@calebschoepp, to elaborate on @jsturtevant's comment, it depends how containerd is executed.
For example, if using systemd, you can update the systemd log level for the containerd service on each node byt adding an environment variable to the service:

systemctl stop containerd
systemctl edit containerd

Then add this to the containerd file that comes up:

[Service]
Environment="RUST_LOG=info"

Restart the service

systemctl restart containerd

[calebschoepp]

@calebschoepp, to elaborate on @jsturtevant's comment, it depends how containerd is executed. For example, if using systemd, you can update the systemd log level for the containerd service on each node byt adding an environment variable to the service:

systemctl stop containerd
systemctl edit containerd

Then add this to the containerd file that comes up:

[Service]
Environment="RUST_LOG=info"

Restart the service

systemctl restart containerd

I'm following that, but I'm specifically curious how containerd works in a K8s environment. In other words how/where would a SpinKube user set RUST_LOG or OTEL_EXPORTER_OTPL_ENDPOINT?

[kate-goldenring]

For a SpinKube user, i'd imagine we'd add the ability to do this through the runtime class manager

[calebschoepp]

Hey folks this is ready for review!

[devigned]

Since there isn't anything that tests this, can you give provide a screenshot of the traces to demonstrate what they look like? I think it would be helpful for folks to see what the traces look like to better understand the value of OTel support in the shim.

[calebschoepp]

Since there isn't anything that tests this, can you give provide a screenshot of the traces to demonstrate what they look like? I think it would be helpful for folks to see what the traces look like to better understand the value of OTel support in the shim.

Great point @devigned. Here's a photo of the traces you would get from a request to a simple Spin app that is setting something in a key value store.

[devigned]

lgtm!