Added Template for Dockerized Equinox Telemetry

CHANGELOG.md

@@ -11,6 +11,7 @@ The `Unreleased` section name is replaced by the expected version of next releas
 ### Added
 
 - Demonstrate `TransactWith` implementation 'pattern' [#113](https://github.com/jet/dotnet-templates/pull/113) 
+- `eqxTelemetry`: Add template for quickly getting dashboards running [#108](https://github.com/jet/dotnet-templates/pull/108)
 
 ### Changed
 

equinox-telemetry/.gitignore

@@ -0,0 +1 @@
+grafana/dashboards/

equinox-telemetry/.template.config/template.json

@@ -0,0 +1,20 @@
+{
+  "$schema": "http://json.schemastore.org/template",
+  "author": "@jet @ragiano215",
+  "classifications": [
+    "CosmosDb",
+    "ChangeFeed",
+    "ChangeFeedProcessor",
+    "Event Sourcing",
+    "Equinox",
+    "Propulsion"
+  ],
+  "tags": {
+    "language": "F#"
+  },
+  "identity": "Equinox.Telemetry",
+  "name": "Equinox Telemetry",
+  "shortName": "eqxTelemetry",
+  "sourceName": "TelemetryTemplate",
+  "preferNameDirectory": true
+}

equinox-telemetry/CONTRIBUTING.md

@@ -0,0 +1,96 @@
+There are 2 core themes in this telemetry template:
+
+* streamlining the experience of developers simply wanting to get telemetry out of the box
+* facilitating developers enhancing the dashboards by establishing a convention
+
+One would find that much of the quirks that drive the developing experience for this template is Grafana's limitations.
+In an ideal world, the most important elements: the panels and / or panel groups should be easily mixed and matched in
+some intuitive fashion based on a few high-level decisions:
+
+* usage of Equinox and / or Propulsion
+* choice of data store: some subset of Cosmos, EventStore, Kafka, etc.
+* additional features that were opted in: `Propulsion.Feed`
+
+The goal is to create our own means to execute this mixing and matching. And the `build-dashboards.py` Python script is
+the culmination of the limited understanding from:
+
+* the [official spec on the Grafana website](https://grafana.com/docs/grafana/v8.3/dashboards/json-model/)
+* observations in how changing the JSON affects the rendering
+
+## Flow
+
+Below is the expected flow for a developer wishing to enhance the dashboards
+
+* using Grafana's facility for provisioning a dashboard based on files in directories
+* load dashboard in UI and make changes in the normal, interactive way (i.e.: not directly against the JSON)
+* copy and paste JSON model (i.e.: spec) into version control
+
+Further
+
+* spec files would split by base Equinox vs base Propulsion dashboards, then data stores
+* any new grouping should lead to extension of the `build-dashboards.py` command line arg parser
+
+> Currently the split is just by Equinox vs. Propulsion.
+>
+> In addition, Python was chosen because it is generally hard to figure what programming runtime comes pre-installed in
+> everyone's machines or where it would require the least effort to get developers wanting to just have telemetry
+> working in an Equinox app set up. Mac and the majority of Linux users reasonably have this pre-installed. Windows
+> users would usually solve this with standard, straightforward installers.
+>
+> Python _3_ was chosen because overall development of the language is sunsetting version 2.
+
+Much of the Python code is visually sectioned with comments and best effort has been made to make every step in
+generating an output JSON understandable.
+
+Below, the main fields driving the JSON that are to be discussed will be around
+
+* panels
+* variables
+
+## Core Aspects Of Quirks
+
+> Brief disclaimer: much of what's articulated is clearly not from someone that's a contributor to the Grafana source
+> code. It'd be swell if the grasping for straws below was updated after being better armed with knowledge from the
+> source code and seeing how JSON specs _really_ translate to what is rendered on the screen.
+
+### Panel Positioning
+
+Let's start with the `panels` key at the root. From what I observed in Grafana 7.X and 8.X, at least, the main 2
+elements for one's mental model are panels and panel groups.
+
+There's a coordinate system based on a grid that is 24 units horizontally. In the vertical direction, the grid extends
+to infinity.
+
+When scripting the programmatic placement of panels onto that grid, one must visualize this 2-dimensional grid with the
+following attributes with distance measured in what we'll just refer to as "units":
+
+* `w` width (value goes up to 24 units, the grid's maximum horizontal length)
+* `h` height
+* `x` x-coordinate on the grid
+* `y` y-coordinate on the grid
+
+Panel groups are simple in that they're expected to take the whole width of the screen (i.e.: `w` equals 24) and always
+have a height of 1 unit. However, because they can either be collapsed or not (i.e.: JSON key of `collapsed` being
+`true` or `false`), only if they're collaped would the member panels sit within a child `panels` key. Otherwise, the
+member panels would exist as sibling JSON to the panel group.
+
+Presently, not much effort has been spent to implement "tight stacking", as it were, for both the horizontal and
+vertical directions. Within reason, it is expected that the JSON that's commited to version control would've had a
+sensible rendering prior to this.
+
+## Dashboard Variables
+
+These sit, starting at the root, at `templating.list`. Each child JSON of this array corresponds to a dashboard variable
+that visually manifests as a dropdown at the top of the dashboard. To date, the only relevance is to conditionally
+render a `group` variable depending on whether Propulsion panels are to show up (i.e.: Equinox doesn't have a notion of
+groups).
+
+## Shared Quirks
+
+Both panels and variables JSONs have a `datasource` key sitting somewhere within them. Notably, in version 7.X of
+Grafana, this would simply be a strong. But 8.X has it as an object. This is currently the cause of friction for
+updating panels while on two different versions of Grafana for development.
+
+# Appendix
+
+* Grafana's provisioning utility: https://grafana.com/docs/grafana/latest/administration/provisioning/

equinox-telemetry/README.md

@@ -0,0 +1,73 @@
+# Equinox Telemetry
+
+This project was generated using:
+
+    dotnet new -i Equinox.Templates # just once, to install/update in the local templates store
+    dotnet new eqxTelemetry
+
+The purpose of this template is to allow a developer to quickly get set up with dashboards while running their Equinox /
+Propulsion application(s) locally. This setup uses Docker to allow one to reproducibly spin up sandboxed instances of
+Prometheus and Grafana that come pre-wired and pre-provisioned with the canonical Equinox dashboards.
+
+Of note: the versions of the dependencies are locked down to:
+
+* Prometheus 2.26.0
+* Grafana 7.5.10
+
+These can be easily changed in the corresponding `Dockerfile`s under the `prometheus` and `grafana` directories.
+
+Additionally, this set-up uses the very simple, Prometheus file-based service discovery mechanism to tell it what
+processes to scrape for metrics. Of note:
+
+* the file `config/targets.yml` specifies what are the target host(s) (in this case, `host.docker.internal` acting as
+  `localhost` in Docker world) and port(s) to be scraped
+* this file has been set up with a Docker bind-mount so that the containerized Prometheus can still see it on the host
+  filesystem and any changes made to it
+* Prometheus uses a file watcher in this approach, so it'll know if new application instances are to be kept track of
+
+## Usage Instructions
+
+The main point of change necessary is to add a line in `config/targets.yml` for the known port of a running instance of
+an Equinox application. Usually, this would be spec-ed via `-p` in the other templates.
+
+Then, run the `build-dashboards.py` Python 3 script (while in the same directory) and specify one or a combo of the
+following to have the dashboard contain the panel groups relevant for your app:
+
+* `--eqx`: to include Equinox panel groups
+* `--prp`: to include Propulsion panel groups
+
+> A help message will be displayed if you execute the script with `--help`.
+
+This will generate the final Grafana spec and place it in the correct directory for Docker to build with.
+
+Finally, while in the same directory as `docker-compose.yml`, simply run the following to start up Prometheus and
+Grafana
+
+```
+docker compose up -d
+
+OR
+
+docker-compose up -d # for older versions of Docker Compose
+```
+
+and connect to `http://localhost:3000` via the browser to access Grafana.
+
+> You may need to run `docker build` first if this is not the first time you ran the above command since the directory
+> contents may have changed (i.e.: you may have initially run the Python script with different options).
+
+After setting the admin password for the first time, the canonical Equinox dashboard can be found under the `equinox`
+directory.
+
+## Side Notes
+
+This simple set-up is not meant for production since the Prometheus store would require a persistent volume. More
+advanced users of Docker can use this template as a starting point and similarly pre-provision their instances with the
+Equinox dashboard spec.
+
+Additionally, `targets.yml` is put in a directory as opposed to being directly bind-mounted to the Prometheus container
+because this approach is not supported on Windows. In fact, it is discouraged even on Linux systems according to
+[this](https://github.com/moby/moby/issues/30555).
+
+This `README.md` is meant for developers trying out Equinox or wanting to get the latest specs for the dashboards. Refer
+to the `CONTRIBUTING.md` if you wish to enhance them and others' telemetry experience.

equinox-telemetry/build-dashboards.py

@@ -0,0 +1,77 @@
+from pathlib import Path
+
+import json
+import argparse
+
+parser = argparse.ArgumentParser(description="Specify dashboard build output.")
+parser.add_argument("--eqx", action="store_true", help="Specify Equinox panel groups to be included.")
+parser.add_argument("--prp", action="store_true", help="Specify Propulsion panel groups to be included.")
+args = parser.parse_args()
+
+flags = vars(parser.parse_args())
+if not any(flags.values()):
+    parser.error("At least one of `--eqx` or `--prp` must be specified.")
+
+# Set baseline dashboard
+with open("dashboards/base.json") as fd:
+    base = json.load(fd)
+    base["panels"] = []
+
+# Step to decide set of panels to include in output
+# NOTE: Current preference is for Propulsion panel groups to be on top
+if args.prp:
+    with open("dashboards/propulsion.json") as prp:
+        prp = json.load(prp)
+        base["panels"] += prp["panels"]
+
+if args.eqx:
+    with open("dashboards/equinox.json") as eqx:
+        eqx = json.load(eqx)
+        base["panels"] += eqx["panels"]
+
+# Set title based on combo specified
+title = "Equinox Metrics" # Default title
+if args.eqx and args.prp:
+    title = "Equinox/Propulsion Metrics"
+elif args.eqx:
+    title = "Equinox Metrics"
+elif args.prp:
+    title = "Propulsion Metrics"
+
+base["title"] = title
+
+# If Propulsion panels are not to be included, remove `group` dropdown
+if not args.prp:
+    ref = base["templating"]["list"]
+    base["templating"]["list"] = list(filter(lambda entry: entry["name"] != "group", ref))
+
+# If have to down-convert to 7.X style data source spec
+for panel in base["panels"]:
+    # NOTE: Conditional `"datasource" in panel"` is needed because it's not consistently there
+    # Reference: https://github.com/grafana/grafana/issues/44506
+    if "datasource" in panel and type(panel["datasource"]) is dict:
+        panel["datasource"] = "$datasource"
+    if panel.get("collapsed", False):
+        for subpanel in panel["panels"]:
+            if type(subpanel["datasource"]) is dict:
+                subpanel["datasource"] = "$datasource"
+
+# Additionally down-convert for dashboard variables
+for variable in base["templating"]["list"]:
+    if "datasource" in variable and type(variable["datasource"]) is dict:
+        variable["datasource"] = "$datasource"
+
+# Adjust y-coordinates / vertical positioning for included panels
+# NOTE: A simplification is used here to accommodate for the fact that more than one panel can take
+# up a row. A deep dive is available in `CONTRIBUTING.md`.
+y_coor = 0
+for index in range(len(base["panels"])):
+    panel = base["panels"][index]
+    panel["gridPos"]["y"] = y_coor
+    if panel["gridPos"]["x"] + panel["gridPos"]["w"] == 24:
+        y_coor += panel["gridPos"]["h"]
+
+# Write out for Docker to build and include for Grafana
+Path("grafana/dashboards").mkdir(exist_ok=True)
+with open("grafana/dashboards/equinox.json", "w") as fd:
+    json.dump(base, fd, indent=2)

equinox-telemetry/config/targets.yml

@@ -0,0 +1,7 @@
+- targets:
+  # UPDATE HERE
+  # For each running process utilizing Equinox and / or Propulsion, add a line here
+  # prefixed by `host.docker.internal` followed by the port number that process is
+  # exposing for Prometheus metrics
+  # e.g.:
+  # - 'host.docker.internal:9000'