Limitations and Evolution of the OpenTelemetry Schema file

[lquerel]

Problem Description

Currently, OTEP 0152 defines a series of transformations aimed at enabling the migration of signals and attributes between two versions of the OTEL semantic convention registry, whether migrating from an older version to a newer one or vice versa. Unfortunately, this idea of bidirectional transformation has proven to be impossible except in the simplest cases.

Consider this scenario:

• Initial state: Both telemetry producers and consumers are using semconv version 1.28.
• New semconv version 1.29: During the transition, there will be a period where some producers and consumers are on version 1.28, while others have migrated to 1.29. As a result, a consumer on version 1.29 might receive data from a 1.28 producer, and vice versa; ideally, both sides would be aligned on the same version, but this isn’t guaranteed.

The idea of reversible transformations seemed promising, as it would theoretically allow:

• A 1.29-compatible consumer to interpret a 1.28 telemetry stream, or
• A 1.28-compatible consumer to convert a 1.29 telemetry stream back into a 1.28-compatible format.

The issue, however, is that these transformations aren’t truly reversible, even in their current, simple forms. The problem arises when transformations consolidate multiple names into a single name (e.g., m1, m2, m3 are unified into a single m4 metric). In such cases, a 1.28-compatible consumer wouldn’t be able to convert back from 1.29 telemetry, as m4 could represent any of m1, m2, or m3 metrics, with no way to reliably determine the original source. This challenge also applies if the transformation renames metrics to an existing name - if, say, m1, m2, and m3 are renamed to m2, it becomes impossible to reverse this reliably.

Another well-known issue is that long-term stored data in an observability backend cannot be unified to the same version at any given moment. We cannot avoid cases where telemetry data is stored with different versions over a long period of time.

Proposed Solutions

Based on our observations, the type of migration described above is very common and cannot be avoided, whether during transport, ingestion, or at the query layer of telemetry data. Therefore, it is essential to provide a solution. We propose two complementary approaches. The first requires minimal ecosystem support and can be implemented relatively quickly. The second approach requires backend support, particularly within query systems.

Short Term (Partial Solution)

By relaxing the requirement to support bidirectional migrations (both backward and forward), it becomes feasible to handle the type of migration previously mentioned effectively and without ambiguity. The idea is to support and document only forward transformations (e.g., 1.28 → 1.29). Specifically, a backend (or collector) configured with semantic conventions 1.29 would be responsible for converting telemetry data in formats 1.26, 1.27, or 1.28 to format 1.29, while any data in 1.29 or later formats would remain unchanged. This is a “best effort” approach where no ambiguity, such as described above, arises.

Long Term (Complete Solution)

To take this a step further, the idea is to enable the query layer to accept the latest version of the semantic convention schema as a parameter (where possible) to perform required transformations on-the-fly, achieving a unified view of telemetry data. To reach this goal, we need to make all schema versions accessible and easy to interpret. Observability backends would also need to store the registry version for each signal and be capable of applying the transformations described in the schema matching each signal’s storage version. This would naturally require integration within these systems.

Next Steps

We propose creating a new OTEP to redefine the scope of transformations described in the existing OTEP 0152. We also suggest adding a link to a resolved version of the semantic convention registry or including a resolved representation of this registry directly within the OTEL Schema file. The choice between these two approaches is yet to be determined. The most immediate priority is to restrict the scope of transformations to support only forward transformations, making this schema usable.

[lquerel]

@ankitpatel96 and @dineshg13 This is another discussion on the topic we discussed this morning.

You should also join the #otel-weaver channel on Slack.

[ancostas]

This requires consumers to support future versions that will be upgraded to, rather than current versions that will be downgraded to, right? That seems infeasible without coupled/lockstep releases.

What is wrong with the current strategy of supporting downgrades? Yes, destructive mappings (i.e. functions that are not one-to-one) are not supported since they are not invertible. But otherwise you can invert the mapping (even if it is not onto; restrict the codomain in version (X+1) to the range of version (X) and now it is onto & 1-1) and map back to the older version. This is something that consumers can easily support without any coupling, given the current scope of Telemetry Schemas.

p.s. I am @ankitpatel96 and @dineshg13's colleague; planning to attend upcoming SIG meetings in case that's a move convenient discussion forum 😁 .