Semantic convention generator

[thisthat]

Ported the code from open-telemetry/opentelemetry-specification#571 into a docker image that is published to docker-hub with the name otel/semconvgen. This image will be later used to automatically generate tables for OpenTelemetry-specifications and code for the different languages.

[bogdandrutu]

@open-telemetry/python-approvers I would like if one of you can review this since I am not that experienced with Python

[aabmass]

I didn't look too deeply into all of the code, it would be great if someone else could also take a look.

I left the same comment on the spec PR (open-telemetry/opentelemetry-specification#571 (comment)), but I am wondering how this will be extended to metric and resource semantic conventions.

A lot of the code/types will be very similar (like attributes to metric labels), but its not clear to me how easy it will be to generalize this to other semantic conventions. Specifically SemanticConvention has class properties that are very specific to spans, and the code generation and markdown table generation don't look reusable yet.

[aabmass]

There's a lot of complex string building in here. Would it be cleaner in a Jinja template?

[thisthat]

Since we are using a markdown linter in the spec repository, I preferred building the string via code to have better control over newlines and spaces. I agree that this could be done using Jinja templates. However, I believe it will just push the complexity from the code to the template.

[Oberon00]

Indeed we had that discussion internally, but usually the more complicated the string building, the less it profits from a Jinja template, which is fine for building large but more or less simple strings. Just imagine how many nesting levels or macros you would have.

[thisthat]

@aabmass thank you for your review :) I have addressed your comments beside the one that I left it unresolved.

I left the same comment on the spec PR (open-telemetry/opentelemetry-specification#571 (comment)), but I am wondering how this will be extended to metric and resource semantic conventions. A lot of the code/types will be very similar (like attributes to metric labels), but its not clear to me how easy it will be to generalize this to other semantic conventions.

As was already said in the spec issue, at Dynatrace we are using the current model for both, resource and span semantic conventions. We did not yet look into metrics, so the model might require some changes to support this part of OTel.

Specifically SemanticConvention has class properties that are very specific to spans, and the code generation and markdown table generation don't look reusable yet.

It is not clear to me what do you consider not reusable yet. The markdown part is tailored for out linter and style used in the specification repository. The code generator uses Jinja templates so any language could use its implementation. This part was already used in the java-instrumentation to generate Typed Spans: open-telemetry/opentelemetry-java-instrumentation#502

[aabmass]

The code generator uses Jinja templates so any language could use its implementation.

I see now the CLI args are flexible, understood :)

As was already said in the spec issue, at Dynatrace we are using the current model for both, resource and span semantic conventions. We did not yet look into metrics, so the model might require some changes to support this part of OTel.

Awesome, I was thinking it would be mostly compatible. Some things I noticed (which I could definitely be wrong about):

• span_kind, optional enum, specifies the kind of the span. Leaf semconv nodes (in the hierarchy tree) that do not have this field set will generate a warning.

  This would need to be relaxed to be compatible with metrics/resources?

• Likewise, there are some extra fields that would be useful to have for metrics. See the tables in OTEP 119, they have "Units" and "Instrument Type". I haven't written the spec PR yet, but I think this is useful to implementors.

• attributes would be a little confusing to see in metrics/resource semantic conventions instead of "labels", but I suppose they are the same thing. (edit: I see only FAAS resources uses "Labels" in the table)

• I believe the markdown output would be slightly different depending on the "type" of semantic convention, e.g. for metrics changing the "Attributes" column to "Labels" and adding output for "Instrument Type" and "Units".

Maybe it would be useful to add a type: 'trace' | 'metrics' | 'resource' field, so that you can verify extends are matching in type, verify fields only specific to a given type (like span_kind, instrument_type, units, labels, attributes), and chose the correct markdown generator or column names. Something like

groups ::= semconv
       | trace_semconv groups
       | metrics_semconv groups
       | resource_semconv groups

# second field is the type literal
trace_semconv ::= id "trace" brief [note] [prefix] [extends] [span_kind] attributes [constraints]
metrics_semconv ::= id "metrics" brief [note] [prefix] [extends] [instrument_type] [units] labels [constraints]
resource_semconv ::= id "resource" brief [note] [prefix] [extends] labels [constraints]


# extends MUST point to an existing semconv id of the same type
extends ::= string

[thisthat]

Awesome, I was thinking it would be mostly compatible. Some things I noticed (which I could definitely be wrong about):

• span_kind, optional enum, specifies the kind of the span. Leaf semconv nodes (in the hierarchy tree) that do not have this field set will generate a warning.
  This would need to be relaxed to be compatible with metrics/resources?

Thank you for noticing the error. I will update the other PR fixing it. This field is optional and will not print any warning. It is already relaxed to support resources :)

Regarding metrics and the split into groups, I agree with your comment and thank you for the feedback :)
I would like to start moving existing semantic conventions in a yaml format so I could also automate the generation of utility code in different SIGs (e.g. Java) that is being a chore to maintain manually.

I would provide a follow-up PR to support Resources and Metrics as first-class citizens.

[bogdandrutu]

@aabmass if python looks good please approve :)

[aabmass]

👍

There's lot of code, so it would be great if someone else could also review the python, maybe @open-telemetry/python-approvers

[bogdandrutu]

Will merge this to unblock the specs