Need a soonish fix for current usage of code generation

[AlexanderWert]

• In the past the way build-tools were used for code generation (in many language SDKs) assumed a strict separation between resource attributes and semantic attributes
• This assumption is not true anymore, as there are attributes that can be used as both, resource and semantic attributes
• Currently, there's no (easy/at all?) way to use the build-tools for code generation in a way that the above separation applies if we have namespaces in the registry that would contain both, resource and semantic attributes
• With the migration to a registry-based attributes model, the current limitation

The above leads to the fact that some work on semantic conventions is blocked, because we cannot introduce namespaces with a mix of resource and semantic attributes.

[AlexanderWert]

\cc @lmolkova FYI: I created this issue to track all ongoing semconv PRs that are blocked by this issue.

[joaopgrassi]

Given #243, do we still need anything else to close this? @AlexanderWert

[lmolkova]

Adding a list of remaining things not yet addressed in #243

Critical, needed to merge to main and release:

• File names should follow per-language pattern (client_attrubutes.py, ClientAttrubutes.java)
• Update PRs in Java and Python (Generate classes by root namespace semantic-conventions-java#45, New semconvgen template and update to semconv 1.25.0 opentelemetry-python#3586) and get green light from Java and Python SIGs

Can be done as separate PRs, not critical for build-tools release:

• Add examples for attributes and metric generation (grouped by namespaces, split by stability, anything else that's common). [update]: done in Add enum_attributes to the context, also adds print_member_value helper #266
• Improve docs (what's available in Jinja template and how to use it to customize things)
• Nits/inconveniences:
  • don't create file if template filtered out all attributes - see this for the details
  • similar: don't import if not necessary - maybe just a doc/sample that shows how to. [update] done in New semconvgen template and update to semconv 1.25.0 opentelemetry-python#3586
  • example on how to generate attributes as module-level variables. [update] done in New semconvgen template and update to semconv 1.25.0 opentelemetry-python#3586
  • automate updating Schema URLs list (see this on python and this on java)
• Ability to generate grouped-by-namespace classes in one file - done, we have new property attributes_and_templates grouped by namespace available on the context when generating everything in one file.

Done, needs approvals:

• Make stability and deprecation independent properties #244

Needs discussion:

• Codegen: proper casing for multiword attribute/metric names semantic-conventions#599 - AFAIK it's not blocking. None of the affected attributes are stable yet

[jsuereth]

I created a prototype to help with filenames: feature/codegen-by-namespace...jsuereth:build-tools:wip-filename-choices

TL;DR; is we can allow --output filename to be a JINJA2 template itself and provide different versions of the "prefix" for the file to the template,e .g. camel-case, snake-case, ....

[joaopgrassi]

I created a prototype to help with filenames: feature/codegen-by-namespace...jsuereth:build-tools:wip-filename-choices

TL;DR; is we can allow --output filename to be a JINJA2 template itself and provide different versions of the "prefix" for the file to the template,e .g. camel-case, snake-case, ....

LGTM

#244

I merged that one now.

[lmolkova]

I believe we have just two items left, but I don't consider them blocking:

• don't create file if template filtered out all attributes - see this for the details:

  • We have means to not create empty files, it's just not super-convenient.
  • If we implemented it, it would not change generated code in any way, but implementation is not trivial.

• automate updating Schema URLs list (see this on python and this on java):

  • This is something we can't achieve with Jinja templates. Also it's easy to do manually (add new constant to the list of schema urls when generating new semconv). Java also added check for it.

So the only thing before we start merging to main is getting green light from Java and Python - I'll check with Python tomorrow.

[jsuereth]

Regarding This point:

• don't create file if template filtered out all attributes - see this for the details

I wasn't sure how to make progress. IIUC, the filter is something applied by the template. Should we make filter more first class in the tooling? Specifically:

• We provide filtered_attributes to the template
• When filtered_attributes is empty, we do not ask to write the file

[lmolkova]

@jsuereth That's what I was also thinking.

Currently filter is fully controlled by the template and python code does not know, so can't prevent empty files.
At the same time, we can make filters first-class citizens (e.g. stable, experimental). This would also save a tiny bit of effort in jinja templates.

If we did this, we could reuse existing --only cmd arg which essentially applies filter on signal. We could change it to accept stability filters like codegen --only metrics,stable or codegen --only experimental.

With codegen --only metrics,stable SIGs would need to run codegen script for each combinations they want.
With codegen --only experimental SIGs would likely have more filters inside the template (for signals) and they'd need to take care of empty files on their own.

I feel this feature (at least how I see it) brings diminishing returns or just needs more thought than I put into it.

[lmolkova]

I believe this can be closed - we got configuration from Java and Python SIGs that codegen works well. If there are any further improvements, we can track them in new issues. Thanks everyone!