Add Otel/STEF project

[tigrannajaryan]

Otel/STEF (Sequential Tabular Encoding Format) is a new data format and network protocol for OpenTelemetry data.

For the target use-cases Otel/STEF outperforms both OTLP and Otel Arrow (phase 1): Otel/STEF is smaller and/or faster.

See stef.md for details: benchmarks and comparisons to other formats, links to prototypes and description of project goals.

[jsuereth]

I believe there is a legitimate need in OpenTelemetry for a stateful protocol (or even something as simple as adding dictionaries to OTLP itself).

As discussed before, I'm supportive of investigating STEF and bringing it to a usable state.

However - I think we need to understand our end-game here. When we evalaute "why not Arrow" or "Why STEF" or "Why not OTLP", I think this proposal is still lacking our primary goal/scope. It has projects, but not implications of delivery of those projects.

I'd like to make sure we align on where this will be used. I called out two areas I think could dramatically improve from some low-level, stateful, efficient protocols. I don't think these are the only targets, but I'm also not sure these were on your radar either.

Let's confirm we agree on end-state then I'm in.

[jsuereth]

I think there should be two additional goals here (possibly one if you phrase it well).

• We should evaluating an efficient file-based protocol for direct-export from API with out-of-band "Collector-style" SDK. That is, imagine an SDK implementation that can serialize events out-of-band quickly and efficiently, and offers more resilience on process-death. This is what I was working my towards with https://github.com/jsuereth/otlp-mmap/ and I think if we invest in a stateless protocol, this should be a use case it can support.
• Providing guidance to eBPF based telemetry extraction. If we are able to define structures and buffers and efficient stateful communication, we might be able to provide a good set of primitives for eBPF based event-extraction (complementary to my first bullet point). This is an avenue I think may be worth exploring.

[tigrannajaryan]

We should evaluating an efficient file-based protocol for direct-export from API with out-of-band "Collector-style" SDK.

That's certainly something we can extend STEF to do. A mmap-ed ring buffer of STEF frames can be that. Since STEF optionally allows full state resets between frames, it essentially has a stateless mode built-in (with resets happening every frame).

Providing guidance to eBPF based telemetry extraction.

I am not sure I understand this one.

[tigrannajaryan]

I think there should be two additional goals here (possibly one if you phrase it well).

I added a more generic goal that says that the project should evaluate additional use-cases.

[jsuereth]

I think I disagree with this. I understand the concern around limiting investment and engineering resources. However - I think you need to better clarify the target use case in this situation. Is this something only the Go SDK would be able to use with a collector? Is this something for just Collector->Collector communication?

That isn't really answered in this proposal, and I think it's critical.

My $.02 is that if we invest in a stateless protocol It should be an Open Standard (e.g. Arrow/parquet) or have an OpenTelemetry use case net well served by existing open standards and OTLP.

Here, you demonstrate that STEF can outperform arrow in efficiency for transmitting telemetry data. Where in open-telemetry do we use that?

TL;DR; this should have a targeted set of use cases for the protocol.

[tigrannajaryan]

Is this something only the Go SDK would be able to use with a collector? Is this something for just Collector->Collector communication?

Technically, nothing prevents any language SDK from having a STEF exporter. I am only making this claim to avoid placing additional burden on language maintainers. Should we (Otel) find it desirable to have STEF protocol support in SDKs, that's certainly doable.

I think the highest ROI is going to be in Collector->Collector or Collector->backend communication and that's why I suggest to start with that.

Can we extend the project to allow the use-cases you mentioned? Absolutely, and I would love to see that happen. I am just intentionally defining the initial scope to be small enough that we can deliver it quickly.

I agree that direct-export use case you mentioned is also very interesting (not for just performance but for crash-resilience reasons).

I am happy to extend the scope if you think you (or some other Otel contributor) could invest time in that extended scope.

[tigrannajaryan]

have an OpenTelemetry use case net well served by existing open standards and OTLP.

This is my motivation. OTLP is inefficient size-wise. Otel Arrow Phase 1 is better size-wise, but even better is possible (as the benchmarks show) and is also quite expensive cpu-wise. STEF significantly advances the performance of our network protocols on multiple dimensions, with relatively small investment.

The first use case is as is described in the proposal: smaller wire size (network cost savings), less cpu consumption (compute cost savings) by the Collector.

Do you think additional explanation is needed for this use case? Or do you want the additional use cases to be described (e.g. the mmap-ed direct-export)?

[jsuereth]

I think we should showcase where stateful protocols are viable (e.g. how does it work through a load balancer?).

While I think stateful protocols have a lot of good use cases, from a broad sense - I have concerns that we may be optimising network overhead at the expense of inflexible network architecture and memory overhead in storing dictionaries from N clients.

I.e. for small, contextual cases this protocol is amazing and should be optimised for such cases. In broad, highly distributed, cases, we may still need to invest in OTLP optimisations.

To be clear - I think STEF shows a lot of potential and is worth investing in. I want to be explicit in this proposal where we think we see the biggest benefit and where the trade-offs fall off. Let's have a target architecture in mind for benchmarks, comparison and 'success'. I think you have that in your head (and were able to elucide when we talked about this), but it's just not written in the proposal.

[tigrannajaryan]

@jsuereth I completely agree with you, we need answers to that. I think this line in the list of goals touches that:

Publish benchmark-justified guidelines on applicability of Otel/STEF vs OTLP vs Otel Arrow.

If this is not enough I can call it out more explicitly.

[tigrannajaryan]

Added more in 88ede99

[yurishkuro]

I am supportive, but would like to see more clarity on the goals, design principles, and trade-offs. For example, this could be explicitly limited to and optimized for a wire transmission protocol at the expense of other usage patterns, but I did not find these goals clearly stated.

[yurishkuro]

Has there been a comparative analysis done of other existing protocols besides Arrow?

[tigrannajaryan]

Benchmarks include comparison with OTLP, Parquet and Otel Arrow. If you think there are other interesting formats to compare to we can add it to the project goals.

[yurishkuro]

I the Spec I did not see a list of design principles for STEF protocol. Does such list exist? What is the protocol optimized for? For instance, this doc illustrates speed and size benchmarks, but do they come with trade-offs? What about memory layout, zero-copy capabilities, ability to append data, efficiency of query execution against the data, etc.?

[tigrannajaryan]

Added design principles here 2aa7487

[tigrannajaryan]

Let me know if you would like more details.

[brancz]

I'm a bit sceptic, Arrow has extension types, how much have they been explored? It sounds like this format is in philosophy 90% identical to Arrow. This is intended as a mostly internal protocol (since only Go is expected to have an implementation), so even the weirder things we've seen in Arrow would be doable (eg. use a binary array and store custom bytes in them).

Much like @yurishkuro, I'd like to see more on tradeoffs.

[tigrannajaryan]

Added some more benchmark results in an appendix https://github.com/open-telemetry/community/blob/24716f9c027d51c793bea404354454434579f89d/projects/stef.md#appendix-a-benchmark-results

[tigrannajaryan]

I'm a bit sceptic, Arrow has extension types, how much have they been explored?

I think this is a question for Otel Arrow SIG.

It sounds like this format is in philosophy 90% identical to Arrow. This is intended as a mostly internal protocol (since only Go is expected to have an implementation), so even the weirder things we've seen in Arrow would be doable (eg. use a binary array and store custom bytes in them).

Correct, in many ways it is similar to Arrow.

Much like @yurishkuro, I'd like to see more on tradeoffs.

Added a section with design principles that explains the tradeoffs.

[reyang]

What about the ecosystem and interoperability?

[jack-berg]

What do you think about updating these to show the result of using OTLP + various compression algorithms. gzip is the default, but I wrote benchmarks for a variety of others and had good results. I wonder how Otel/STEF And OTEL Arrow + Stream Mode stack up.

[austinlparker]

Heya, super interesting stuff. I have a couple of questions as a GC member --

1. Could you help me understand how this project fits into the project goals, overall? I appreciate that 'stateful OTLP' is useful for many reasons, and I do see that this offers improvements vs. OTLP/Arrow, but given the balance of work that we're trying to tackle in 2025 I'm trying to understand where this fits in.
2. Is the existence of this a result of OTLP/Arrow not being suitable? I agree with some points upthread (e.g., having a stateful 'collector-like' sdk buffer), but it also feels like this being scoped down to just Go/Collector would make it more difficult to achieve that goal.

Appreciate any responses to these points.