add OpenTelemetry instrumentation to titiler.core

[hrodmn]

From #1167

This adds some convenience functions for adding opentelemetry traces to titiler applications in titiler.core.telemetry. These features will only be activated if the opentelemetry dependencies are installed.

When enabled in a titiler application, the traces can be browsed in a UI like Jaeger:

Factory class instrumentation: the BaseTilerFactory has a new method add_telemetry() which will add a trace to each registered endpoint in the factory class.

Endpoint instrumentation: endpoint methods in all of the tiler factory classes in titiler.core, titiler.mosaic, and titiler.xarray have been decorated with some operation_tracer context managers to track the performance of specific operations within an endpoint function.

To keep things clean I opted to skip adding additional attributes to the traces, but we might want to revisit that in the future!

TODO

• add telemetry.py
• decorate factory classes
• add tests
• update docs

[github-actions]

⚠️ Performance Alert ⚠️

Possible performance regression was detected for benchmark 'TiTiler performance Benchmarks'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.30.

Benchmark suite	Current: df331c6	Previous: 995e833	Ratio
WGS1984Quad longest_transaction	0.07 s	0.05 s	1.40

This comment was automatically generated by workflow using github-action-benchmark.

[vincentsarago]

Thanks @hrodmn this looks super promising 🙏

I've added couple comments (mostly on naming).

My main concern is about the number of lines the tracing will add to the factory, mostly due to the set_attribute blocks which we should try to avoid if the information can be derived from other place (e.g by looking at the query parameter, like for dataset path and tile matrix set)

If we move forward, we would need to add trace to all other factory endpoints 😬

[hrodmn]

My main concern is about the number of lines the tracing will add to the factory, mostly due to the set_attribute blocks which we should try to avoid if the information can be derived from other place (e.g by looking at the query parameter, like for dataset path and tile matrix set)

Yes, I agree about not wanting to add so many lines just for tracing. I will work on ways to make attribute setting more succinct. User input parameters should be easy to absorb automatically but anything else would need to be added manually. Maybe we can leave ImageData introspection out for now and add it to rio-tiler if users want it.

If we move forward, we would need to add trace to all other factory endpoints 😬

Thanks to the add_telemetry method we will get basic tracing capability for any factory that is based on BaseFactory for free! If we want any of the more detailed within-function traces (like open_dataset, read_tile, etc) that will require changes in every method definition where we need more detailed info.

[hrodmn]

We could decide to completely skip setting attributes for and just add the operation_tracer context manager in places where we want more detailed profiling or performance evaluation.

We can still get a trace timeline like this without exporting all of the extra attributes:

[hrodmn]

@vincentsarago thanks for your feedback on these changes! I have implemented your naming suggestions and trimmed out the verbose attribute tracing lines.

I did some research on the expected performance hit for including these traces and I don't think it will be significant. Using the BatchSpanProcessor reduces overhead for an application because it collects traces in batches and sends them to the collector endpoint in a separate thread. It is also conventional for an application to only collect traces for a sample (e.g. 2%) of requests, which would limit the impact of enabling telemetry if there is any measureable performance reduction.

[geospatial-jeff]

I did some research on the expected performance hit for including these traces and I don't think it will be significant.

Historically OTEL has had some bad performance issues but most of that has been worked out over the last few years 👍

[geospatial-jeff]

🚀