Allow commit time exporter to use information from annotations.

Change to resolve dora-metrics#317.

The idea is taken from PR dora-metrics#381, however the implementation
allows to have minimal required annotations to calculate
commit time. It also allows to define custom annotations.

Co-authored-by: Mike Hepburn <eformat@gmail.com>

charts/pelorus/Chart.yaml

@@ -14,7 +14,7 @@ type: application
 
 # This is the chart version. This version number should be incremented each time you make changes
 # to the chart and its templates, including the app version.
-version: 1.6.1
+version: 1.6.3
 
 dependencies:
  - name: exporters

charts/pelorus/configmaps/committime.yaml

@@ -13,3 +13,5 @@ data:
  GIT_PROVIDER: "default" # github | github, gitlab, or bitbucket
  TLS_VERIFY: "default" # True
  NAMESPACES: # | Restricts the set of namespaces, comma separated value "myapp-ns-dev,otherapp-ci"
+ COMMIT_HASH_ANNOTATION: "default" # io.openshift.build.commit.id | commit hash annotation name associated with the Build
+ COMMIT_REPO_URL_ANNOTATION: "default" # io.openshift.build.source-location | commit repository URL annotation name associated with the Build

docs/Configuration.md

@@ -95,6 +95,22 @@ We require that all builds associated with a particular application be labelled
 
 Currently we support GitHub and GitLab, with BitBucket coming soon. Open an issue or a pull request to add support for additional Git providers!
 
+#### Annotated Binary (local) source build support
+
+Commit Time Exporter may be used in conjunction with Builds where values required to gather commit time from the source repository are missing. In such case each Build is required to be annotated with two values allowing Commit Time Exporter to calculate metric from the Build.
+
+To annotate Build use the following commands:
+
+```shell
+oc annotate build <build-name> -n <namespace> --overwrite io.openshift.build.commit.id=<commit_hash>
+
+oc annotate build <build-name> -n <namespace> --overwrite io.openshift.build.source-location=<repo_uri>
+```
+
+Custom Annotation names may also be configured using ConfigMap Data Values.
+
+Note: The requirement to label the build with `app.kubernetes.io/name=<app_name>` for the annotated Builds applies.
+
 #### Suggested Secrets
 
 Create a secret containing your Git username and token.
@@ -137,6 +153,9 @@ This exporter provides several configuration options, passed via `pelorus-config
 | `APP_LABEL` | no | Changes the label key used to identify applications | `app.kubernetes.io/name` |
 | `NAMESPACES` | no | Restricts the set of namespaces from which metrics will be collected. ex: `myapp-ns-dev,otherapp-ci` | unset; scans all namespaces |
 | `PELORUS_DEFAULT_KEYWORD` | no | ConfigMap default keyword. If specified it's used in other data values to indicate "Default Value" should be used | `default` |
+| `COMMIT_HASH_ANNOTATION` | no | Annotation name associated with the Build from which hash is used to calculate commit time | `io.openshift.build.commit.id` |
+| `COMMIT_REPO_URL_ANNOTATION` | no | Annotation name associated with the Build from which GIT repository URL is used to calculate commit time | `io.openshift.build.source-location` |
+
 
 ### Deploy Time Exporter
 

exporters/committime/README.md

@@ -21,9 +21,38 @@ This exporter currently pulls build data from the following systems:
  * Docker builds
  * JenkinsPipelineStrategy builds
 
+* OpenShift - We look for `Build` resources with `Annotations` where `.spec.source.git.uri` and `.spec.revision.git.commit` were missing. This includes:
+ * Binary (local) source build
+ * Any build type
+
 Then we get commit data from the following systems through their respective APIs:
 
 * GitHub
 * GitHub Enterprise (including private endpoints)
 * Bitbucket _(coming soon)_
 * Gitlab _(coming soon)_
+
+## Annotated Binary (local) source build support
+
+OpenShift binary builds are a popular mechanism for building container images on OpenShift, where the source code is being streamed from a local file system to the builder.
+
+These type of builds do not contain source code information, however you may annotate the build phase with the following annotations for pelorus to use:
+
+| Annotation | Example | Description |
+|:-|:-|:-|
+| `io.openshift.build.commit.id` | cae392a | Short or Long hash of the source commit used in the build |
+| `io.openshift.build.source-location` | https://github.com/org/myapp.git | Source URL for the build |
+
+Annotations for the hash and source-location may have different names. Configuration for such annotation is configurable via ConfigMap for the committime exporter. Example:
+
+`COMMIT_HASH_ANNOTATION="io.custom.build.commit.id"`
+
+`COMMIT_REPO_URL_ANNOTATION="io.custom.build.repo_url"`
+
+Example command to put in your build pipeline each time you start an OpenShift `Build`:
+
+```sh
+oc annotate bc/${BUILD_CONFIG_NAME} --overwrite \
+ io.openshift.build.commit.id=${GIT_COMMIT} \
+ io.openshift.build.source-location=${GIT_URL} \
+```

exporters/committime/__init__.py

@@ -1,3 +1,22 @@
+#!/usr/bin/env python3
+#
+# Copyright Red Hat
+#
+# Licensed under the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License. You may obtain
+# a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations
+# under the License.
+#
+
+from __future__ import annotations
+
 import logging
 from typing import Optional
 
@@ -14,6 +33,7 @@
 @attr.define
 class CommitMetric:
  name: str = attr.field()
+ annotations: dict = attr.field(default=None, kw_only=True)
  labels: dict = attr.field(default=None, kw_only=True)
  namespace: Optional[str] = attr.field(default=None, kw_only=True)
 
@@ -98,19 +118,28 @@ def __parse_repourl(self):
 
  # maps attributes to their location in a `Build`.
  #
- # missing fields are handled specially:
+ # missing attributes or with False argument are handled specially:
+ #
  # name: set when the object is constructed
  # labels: must be converted from an `openshift.dynamic.ResourceField`
  # repo_url: if it's not present in the Build, fallback logic needs to be handled elsewhere
+ # commit_hash: if it's missing in the Build, fallback logic needs to be handled elsewhere
  # commit_timestamp: very special handling, the main purpose of each committime collector
+ # comitter: not required to calculate committime
  _BUILD_MAPPING = dict(
- build_name="metadata.name",
- build_config_name="metadata.labels.buildconfig",
- namespace="metadata.namespace",
- commit_hash="spec.revision.git.commit",
- committer="spec.revision.git.author.name",
- image_location="status.outputDockerImageReference",
- image_hash="status.output.to.imageDigest",
+ build_name=["metadata.name", True],
+ build_config_name=["metadata.labels.buildconfig", True],
+ namespace=["metadata.namespace", True],
+ image_location=["status.outputDockerImageReference", True],
+ image_hash=["status.output.to.imageDigest", True],
+ commit_hash=["spec.revision.git.commit", False],
+ repo_url=["spec.source.git.uri", False],
+ committer=["spec.revision.git.author.name", False],
+ )
+
+ _ANNOTATION_MAPPIG = dict(
+ repo_url="io.openshift.build.source-location",
+ commit_hash="io.openshift.build.commit.id",
  )
 
 
@@ -124,8 +153,8 @@ def commit_metric_from_build(app: str, build, errors: list) -> CommitMetric:
  # Collect all errors to be reported at once instead of failing fast.
  metric = CommitMetric(app)
  for attr_name, path in CommitMetric._BUILD_MAPPING.items():
- with collect_bad_attribute_path_error(errors):
- value = get_nested(build, path, name="build")
+ with collect_bad_attribute_path_error(errors, path[1]):
+ value = get_nested(build, path[0], name="build")
  setattr(metric, attr_name, value)
 
  return metric

exporters/committime/collector_base.py

@@ -1,3 +1,20 @@
+#!/usr/bin/env python3
+#
+# Copyright Red Hat
+#
+# Licensed under the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License. You may obtain
+# a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations
+# under the License.
+#
+
 from __future__ import annotations
 
 import logging
@@ -10,7 +27,12 @@
 
 import pelorus
 from committime import CommitMetric, commit_metric_from_build
-from pelorus.utils import get_nested
+from pelorus.utils import get_env_var, get_nested
+
+# Custom annotations env for the Build
+# Default ones are in the CommitMetric._ANNOTATION_MAPPIG
+COMMIT_HASH_ANNOTATION_ENV = "COMMIT_HASH_ANNOTATION"
+COMMIT_REPO_URL_ANNOTATION_ENV = "COMMIT_REPO_URL_ANNOTATION"
 
 
 class AbstractCommitCollector(pelorus.AbstractPelorusExporter):
@@ -180,16 +202,16 @@ def get_metric_from_build(self, build, app, namespace, repo_url):
  if not self._is_metric_ready(namespace, metric, build):
  return None
 
- if repo_url:
- metric.repo_url = repo_url
- elif get_nested(build, "spec.source.git", default=None):
- metric.repo_url = get_nested(build, "spec.source.git.uri", name="build")
- else:
- metric.repo_url = self._get_repo_from_build_config(build)
-
+ # Populate annotations and labels required by
+ # subsequent _set_ functions.
+ metric.annotations = vars(build.metadata.annotations)
  metric.labels = vars(build.metadata.labels)
 
- metric = self._get_commit_hash(metric, errors)
+ metric = self._set_repo_url(metric, repo_url, build, errors)
+
+ metric = self._set_commit_hash(metric, errors)
+
+ metric = self._set_commit_timestamp(metric, errors)
 
  if errors:
  msg = (
@@ -216,6 +238,65 @@ def get_metric_from_build(self, build, app, namespace, repo_url):
  logging.error(e, exc_info=True)
  return None
 
+ def _set_commit_hash(self, metric: CommitMetric, errors: list) -> CommitMetric:
+ if not metric.commit_hash:
+ commit_hash_annotation = get_env_var(
+ COMMIT_HASH_ANNOTATION_ENV,
+ CommitMetric._ANNOTATION_MAPPIG.get("commit_hash"),
+ )
+ commit_hash = metric.annotations.get(commit_hash_annotation)
+ if commit_hash:
+ metric.commit_hash = commit_hash
+ logging.debug(
+ "Commit hash for build %s provided by '%s' annotation: %s",
+ metric.build_name,
+ commit_hash_annotation,
+ metric.commit_hash,
+ )
+ else:
+ errors.append("Couldn't get commit hash")
+ return metric
+
+ def _set_repo_url(
+ self, metric: CommitMetric, repo_url: str, build, errors: list
+ ) -> CommitMetric:
+ # Logic to get repo_url, first conditon wins
+ # 1. Gather repo_url from the build from spec.source.git.uri
+ # 2. Check if repo_url was passed to the function and use it
+ # 3. Get repo_url from annotations
+ # 4. Get repo_url from parent BuildConfig
+
+ if metric.repo_url:
+ logging.debug(
+ "Repo URL for build %s provided by '%s': %s",
+ metric.build_name,
+ CommitMetric._BUILD_MAPPING.get("repo_url")[0],
+ metric.repo_url,
+ )
+ elif repo_url:
+ metric.repo_url = repo_url
+ else:
+ repo_url_annotation = get_env_var(
+ COMMIT_REPO_URL_ANNOTATION_ENV,
+ CommitMetric._ANNOTATION_MAPPIG.get("repo_url"),
+ )
+ repo_from_annotation = metric.annotations.get(repo_url_annotation)
+ if repo_from_annotation:
+ metric.repo_url = repo_from_annotation
+ logging.debug(
+ "Repo URL for build %s provided by '%s' annotation: %s",
+ metric.build_name,
+ repo_url_annotation,
+ metric.repo_url,
+ )
+ else:
+ metric.repo_url = self._get_repo_from_build_config(build)
+
+ if not metric.repo_url:
+ errors.append("Couldn't get repo_url")
+
+ return metric
+
  def _is_metric_ready(self, namespace: str, metric: CommitMetric, build) -> bool:
  """
  Determine if a build is ready to be examined.
@@ -251,7 +332,7 @@ def _is_metric_ready(self, namespace: str, metric: CommitMetric, build) -> bool:
 
  # TODO: be specific about the API modifying in place or returning a new metric.
  # Right now, it appears to do both.
- def _get_commit_hash(self, metric: CommitMetric, errors: list) -> CommitMetric:
+ def _set_commit_timestamp(self, metric: CommitMetric, errors: list) -> CommitMetric:
  """
  Check the cache for the commit_time.
  If absent, call the API implemented by the subclass.

exporters/pelorus/utils.py

@@ -1,3 +1,21 @@
+#!/usr/bin/env python3
+#
+# Copyright Red Hat
+#
+# Licensed under the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License. You may obtain
+# a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations
+# under the License.
+#
+
+
 """
 Module utils contains helper utilities for common tasks in the codebase.
 They are mainly to help with type information and to deal with data structures
@@ -126,14 +144,16 @@ def __str__(self):
 
 
 @contextlib.contextmanager
-def collect_bad_attribute_path_error(error_list: list):
+def collect_bad_attribute_path_error(error_list: list, append: bool = True):
  """
- If a BadAttributePathError is raised, append it to the list and continue.
+ If a BadAttributePathError is raised, append it to the list of errors and continue.
+ If append is set to False then error will not be appended to the list of errors.
  """
  try:
  yield
  except BadAttributePathError as e:
- error_list.append(e)
+ if append:
+ error_list.append(e)
 
 
 class SpecializeDebugFormatter(logging.Formatter):