Support codegen by namespace (#243)

Co-authored-by: Armin Ruech <7052238+arminru@users.noreply.github.com>
Co-authored-by: Alexander Wert <AlexanderWert@users.noreply.github.com>

.github/workflows/semconvgen.yml

@@ -13,7 +13,7 @@ jobs:
   tests:
     runs-on: ubuntu-latest
     defaults:
-      run: 
+      run:
         working-directory: semantic-conventions/
     steps:
     - uses: actions/checkout@v3
@@ -39,7 +39,7 @@ jobs:
       run: pylint *.py src/
     - name: Type checking (mypy)
       run: mypy src/
-      
+
 
   build-and-publish-docker:
     runs-on: ubuntu-latest
@@ -68,3 +68,13 @@ jobs:
         else
           tag_and_push "${GITHUB_REF#"refs/tags/"}"
         fi
+    - name: Push the Dev Docker image
+      if: startsWith(github.ref, 'refs/heads/feature/')
+      run: |
+        echo "${{ secrets.DOCKER_PASSWORD }}" | docker login -u "${{ secrets.DOCKER_USERNAME }}" --password-stdin
+        function tag_and_push {
+          docker tag semconvgen "otel/semconvgen:${1}" && docker push "otel/semconvgen:${1}"
+        }
+        TAG="${GITHUB_REF#"refs/heads/"}"
+        TAG="${TAG/"/"/"-"}"
+        tag_and_push "${TAG}"

semantic-conventions/CHANGELOG.md

@@ -4,6 +4,11 @@ Please update the changelog as part of any significant pull request.
 
 ## Unreleased
 
+- Added code-generation mode that groups attributes by the root namespace and ability to write each group into individual file.
+  [BREAKING] The `--file-per-group <pattern>` that used to create multiple directories (like `output/<pattern>/file`) now generates
+  multiple files (`output/<pattern>file`) instead.
+  ([#243](https://github.com/open-telemetry/build-tools/pull/243))
+
 ## v0.23.0
 
 - Rephrase and relax sampling-relevant description
@@ -17,7 +22,7 @@ Please update the changelog as part of any significant pull request.
   ([#205](https://github.com/open-telemetry/build-tools/pull/205))
 - Fix referencing template attributes
   ([#206](https://github.com/open-telemetry/build-tools/pull/206))
-  
+
 ## v0.21.0
 
 - Render template-type attributes from yaml files

semantic-conventions/README.md

@@ -85,8 +85,8 @@ convention that have the tag `network`.
 `<!-- semconv http.server(tag=network, full) -->` will print the constraints and attributes of both `http` and `http.server`
 semantic conventions that have the tag `network`.
 
-`<!-- semconv metric.http.server.active_requests(metric_table) -->` will print a table describing a single metric 
-`http.server.active_requests`. 
+`<!-- semconv metric.http.server.active_requests(metric_table) -->` will print a table describing a single metric
+`http.server.active_requests`.
 
 ## Code Generator
 
@@ -125,12 +125,70 @@ This way, multiple files are generated. The value of `pattern` can be one of the
 - `semconv_id`: The id of the semantic convention.
 - `prefix`: The prefix with which all attributes starts with.
 - `extends`: The id of the parent semantic convention.
+- `root_namespace`: The root namespace of attribute to group by.
 
 Finally, additional value can be passed to the template in form of `key=value` pairs separated by
 comma using the `--parameters [{key=value},]+` or `-D` flag.
 
 ### Customizing Jinja's Whitespace Control
 
-The image also supports customising
+The image also supports customizing
 [Whitespace Control in Jinja templates](https://jinja.palletsprojects.com/en/3.1.x/templates/#whitespace-control)
 via the additional flag `--trim-whitespace`. Providing the flag will enable both `lstrip_blocks` and `trim_blocks`.
+
+### Accessing Semantic Conventions in the template
+
+When template is processed, it has access to a set of variables that depends on the `--file-per-group` value (or lack of it).
+You can access properties of these variables and call Jinja or Python functions defined on them.
+
+#### Single file (no `--file-per-group` pattern is provided)
+
+Processes all parsed semantic conventions
+
+- `semconvs` - the dictionary containing parsed `BaseSemanticConvention` instances with semconv `id` as a key
+- `attributes_and_templates` - the dictionary containing all attributes (including template ones) grouped by their root namespace.
+  Each element in the dictionary is a list of attributes that share the same root namespace. Attributes that don't have a namespace
+  appear under `""` key.
+- `attributes` - the list of all attributes from all parsed semantic conventions. Does not include template attributes.
+- `attribute_templates` - the list of all attribute templates from all parsed semantic conventions.
+
+#### The `root_namespace` pattern
+
+Processes a single namespace and is called for each namespace detected.
+
+- `attributes_and_templates` - the list containing all attributes (including template ones) in the given root namespace.
+- `root_namespace` - the root namespace being processed.
+
+#### Other patterns
+
+Processes a single pattern value and is called for each distinct value.
+
+- `semconv` - the instance of parsed `BaseSemanticConvention` being processed.
+
+### Filtering and mapping
+
+Jinja templates has a notion of [filters](https://jinja.palletsprojects.com/en/2.11.x/templates/#list-of-builtin-filters) allowing to transform objects or filter lists.
+
+Semconvgen supports following additional filters to simplify common operations in templates.
+
+#### `SemanticAttribute` operations
+
+1. `is_definition` - Checks if the attribute is the original definition of the attribute and not a reference.
+2. `is_deprecated` - Checks if the attribute is deprecated. The same check can also be done with `(attribute.stability | string())  == "StabilityLevel.DEPRECATED"`
+3. `is_experimental` - Checks if the attribute is experimental. The same check can also be done with `(attribute.stability | string())  == "StabilityLevel.EXPERIMENTAL"`
+4. `is_stable` - Checks if the attribute is experimental. The same check can also be done with `(attribute.stability | string())  == "StabilityLevel.STABLE"`
+5. `is_template` - Checks if the attribute is a template attribute.
+
+#### String operations
+
+1. `first_up` - Upper-cases the first character in the string. Does not modify anything else
+2. `regex_replace(text, pattern, replace)` - Makes regex-based replace in `text` string using `pattern``
+3. `to_camelcase` - Converts a string to camel case (using `.` and `_` as words delimiter in the original string).
+   The first character of every word is upper-cased, other characters are lower-cased. E.g. `foo.bAR_baz` becomes `fooBarBaz`
+4. `to_const_name` - Converts a string to Python or Java constant name (SNAKE_CASE) replacing `.` or `-` with `_`. E.g.
+   `foo.bAR-baz` becomes `FOO_BAR_BAZ`.
+5. `to_doc_brief` - Trims whitespace and removes dot at the end. E.g. ` Hello world.\t` becomes `Hello world`
+
+#### `BaseSemanticConvention` operations
+
+1. `is_metric` - Checks if semantic convention describes a metric.

semantic-conventions/src/opentelemetry/semconv/main.py

@@ -129,7 +129,8 @@ def add_code_parser(subparsers):
     parser.add_argument(
         "--output",
         "-o",
-        help="Specify the output file for the code generation.",
+        help="Specify the output file name for the code generation. "
+        "See also `--file-per-group` on how to generate multiple files.",
         type=str,
         required=True,
     )
@@ -143,8 +144,10 @@ def add_code_parser(subparsers):
     parser.add_argument(
         "--file-per-group",
         dest="pattern",
-        help="Each Semantic Convention is processed by the template and store in a different file. PATTERN is expected "
-        "to be the name of a SemanticConvention field and is prepended as a prefix to the output argument",
+        help="Semantic conventions are processed by the template and stored in a different file. "
+        "File names start with a 'pattern' and end with the name specified in the 'output' argument. "
+        "The 'pattern' can either match 'root_namespace' to group attributes by the root namespace or "
+        "match a name of Semantic Convention property which value will be used as a file name prefix.",
         type=str,
     )
     parser.add_argument(

semantic-conventions/src/opentelemetry/semconv/model/semantic_attribute.py

@@ -60,6 +60,7 @@ class SemanticAttribute:
     sampling_relevant: bool
     note: str
     position: List[int]
+    root_namespace: str
     inherited: bool = False
     imported: bool = False
 
@@ -203,6 +204,10 @@ def parse(
             fqn = fqn.strip()
             parsed_brief = TextWithLinks(brief.strip() if brief else "")
             parsed_note = TextWithLinks(note.strip())
+
+            namespaces = fqn.split(".")
+            root_namespace = namespaces[0] if len(namespaces) > 1 else ""
+
             attr = SemanticAttribute(
                 fqn=fqn,
                 attr_id=attr_id,
@@ -218,6 +223,7 @@ def parse(
                 sampling_relevant=sampling_relevant,
                 note=parsed_note,
                 position=position,
+                root_namespace=root_namespace,
             )
             if attr.fqn in attributes:
                 position = position_data[list(attribute)[0]]
@@ -318,7 +324,6 @@ def parse_stability_deprecated(stability, deprecated, position_data):
 
     @staticmethod
     def check_stability(stability_value, position):
-
         stability_value_map = {
             "deprecated": StabilityLevel.DEPRECATED,
             "experimental": StabilityLevel.EXPERIMENTAL,

semantic-conventions/src/opentelemetry/semconv/model/semantic_convention.py

@@ -264,6 +264,9 @@ def __init__(self, group):
         self.metric_name = group.get("metric_name")
         self.unit = group.get("unit")
         self.instrument = group.get("instrument")
+
+        namespaces = self.metric_name.split(".")
+        self.root_namespace = namespaces[0] if len(namespaces) > 1 else ""
         self.validate()
 
     def validate(self):