From 8bf696948f208cbfef56624b7eacf95641edfa1a Mon Sep 17 00:00:00 2001 From: Josh Suereth Date: Mon, 8 Mar 2021 09:32:57 -0500 Subject: [PATCH 01/14] Initial cut at migrating Josh MacD's datamodel document into the specification. --- specification/metrics/datamodel.md | 184 +++++++++++++++++++++++++++++ specification/overview.md | 10 +- 2 files changed, 192 insertions(+), 2 deletions(-) create mode 100644 specification/metrics/datamodel.md diff --git a/specification/metrics/datamodel.md b/specification/metrics/datamodel.md new file mode 100644 index 00000000000..2f9f1255fd2 --- /dev/null +++ b/specification/metrics/datamodel.md @@ -0,0 +1,184 @@ +# Introduction + +TODO(jsuereth): Write this + +## Events → Data → Timeseries + +The OTLP Metrics protocol is designed as a standard for transporting metric +data. To describe the intended use of this data and the associated semantic +meaning, OpenTelemetry metric data types will be linked into a framework +containing a higher-level model, about Metrics APIs and discrete input values, +and a lower-level model, defining the Timeseries and discrete output values. +The relationship between models is displayed in the diagram below. + +TODO - Diagram + +This protocol was designed to meet the requirements of the OpenCensus Metrics +system, particularly to meet its concept of Metrics Views. Views are +accomplished in the OpenTelemetry Metrics data model through support for data +transformation on the collection path. + +OpenTelemetry has identified three kinds of semantics-preserving Metric data +transformation that are useful in building metrics collection systems as ways of +controlling cost, reliability, and resource allocation. The OpenTelemetry +Metrics data model is designed to support these transformations both inside an +SDK as the data originates, or as a reprocessing stage inside the OpenTelemetry +collector. These transformations are: + +1. Temporal reaggregation: Metrics that are collected at a high-frequency can be + re-aggregated into longer intervals, allowing low-resolution timeseries to be + pre-calculated or used in place of the original metric data. +2. Spatial reaggregation: Metrics that are produced with unwanted dimensions can + be re-aggregated into metrics having fewer dimensions. +3. Delta-to-Cumulative: Metrics that are input and output with Delta temporality + unburden the client from keeping high-cardinality state. The use of deltas + allows downstream services to bear the cost of conversion into cumulative + timeseries, or to forego the cost and calculate rates directly. + +OpenTelemetry Metrics data points are designed so that these transformations can +be applied automatically to points of the same type, subject to conditions +outlined below. Every OTLP data point has an intrinsic +[decomposable aggregate function](https://en.wikipedia.org/wiki/Aggregate_function#Decomposable_aggregate_functions) +making it semantically well-defined to merge data points across both temporal +and spatial dimensions. Every OTLP data point also has two meaningful timestamps +which, combined with intrinsic aggregation, make it possible to carry out the +standard metric data transformations for each of the model’s basic points while +ensuring that the result carries the intended meaning. + +As in OpenCensus Metrics, metrics data can be transformed into one or more +Views, just by selecting the aggregation interval and the desired dimensions. +One stream of OTLP data can be transformed into multiple timeseries outputs by +configuring different Views, and the required Views processing may be applied +inside the SDK or by an external collector. + +### Example Use-cases + +TODO(jsuereth): Flesh this out + +1. OTel SDK exports 10 second resolution to a single OTel collector, using + cumulative temporality for a stateful client, stateless server: + - Collector passes-through original data to an OTLP destination + - Collector re-aggregates into longer intervals without changing dimensions + - Collector re-aggregates into several distinct views, each with a subset of + the available dimensions, outputs to the same destination +2. OTel SDK exports 10 second resolution to a single OTel collector, using delta + temporality for a stateless client, stateful server: + - Collector re-aggregates into 60 second resolution + - Collector converts delta to cumulative temporality +3. A number of OTel SDKs running locally each exports 10 second resolution, each + reports to a single (local) OTel collector. + - Collector re-aggregates into 60 second resolution + - Collector re-aggregates to eliminate the identity of individual SDKs (e.g., + distinct `service.instance.id` values) + - Collector outputs to an OTLP destination +4. Pool of OTel collectors receive OTLP and export Prometheus Remote Write + - Collector joins service discovery with metric resources + - Collector computes “up”, staleness marker + - Collector applies a distinct external label +5. OTel collector receives Statsd and exports OTLP + - With delta temporality: stateless collector + - With cumulative temporality: stateful collector + +These are considered the "core" use-cases used to analyze tradeoffs and design +decisions within the metrics data model. + +# Model Details + +TODO(jsuereth): Recap split of 3 pieces + +## Event Model + +This specification uses as its foundation a +[Metrics API consisting of 6 model instruments](api.md), each having distinct +semantics, that were prototyped in several OpenTelemetry SDKs between July 2019 +and June 2020. The model instruments and their specific use-cases are meant to +anchor our understanding of the OpenTelemetry data model and are divided into +three categories: + +- Synchronous vs. Asynchronous. The act of calling a Metrics API in a + synchronous context means the application calls the SDK, typically having + associated trace context and baggage; an Asynchronous instrument is called at + collection time, through a callback, and lacks context. +- Adding vs. Grouping. Whereas adding instruments express a sum, grouping + instruments characterize a group of measurements. The numbers passed to adding + instruments define division, in the algebraic sense, while the numbers passed + to grouping instruments are generally not. Adding instrument values are always + parts of a sum, while grouping instrument values are individual measurements. +- Monotonic vs. Non-Monotonic. The adding instruments are categorized by whether + the derivative of the quantity they express is non-negative. Monotonic + instruments are primarily useful for monitoring a rate value, whereas + non-monotonic instruments are primarily useful for monitoring a total value. + +In this model, the primary data are (instrument, number) points, originally +observed in real time or on demand (for the synchronous and asynchronous cases, +respectively). The instruments and model use-cases will be described in greater +detail as we link this model with the other two. + + +## Timeseries Model + +In this low-level metrics data model, a Timeseries is defined by an entity +consisting of several metadata properties: + +- Metric name and description +- Label set +- Kind of point +- Unit of measurement + +The primary data of each timeseries are ordered (timestamp, value) points, for +three value types: + +1. Counter (Monotonic, cumulative) +2. Gauge +3. Histogram + +This model may be viewed as an idealization of +[Prometheus Remote Write](https://docs.google.com/document/d/1LPhVRSFkGNSuU1fBd81ulhsCPR4hkSZyyBj1SZ8fWOM/edit#heading=h.3p42p5s8n0ui). +Like that protocol, we are additionally concerned with knowing when a point +value is defined, as compared with being implicitly or explicitly absent. There +is no delta temporality in the timeseries model. To precisely define presence +and absence of data requires further development of the correspondence between +these models. + +## OpenTelemetry data model + +The OpenTelemetry data model for metrics includes four basic point kinds, all of +which satisfy the requirements above, meaning they define a decomposable +aggregate function (also known as a “natural merge” function) for points of the +same kind. The basic point kinds are: + +1. Monotonic Sum +2. Non-Monotonic Sum +3. Gauge +4. Histogram + +Comparing the OpenTelemetry and Timeseries data models, OTLP carries an +additional kind of point. Whereas an OTLP Monotonic Sum point translates into a +Timeseries Counter point, and an OTLP Histogram point translates into a +Timeseries Histogram point, there are two OTLP data points that become Gauges +in the Timeseries model: the OTLP Non-Monotonic Sum point and OTLP Gauge point. + +The two points that become Gauges in the Timeseries model are distinguished by +their built in aggregate function, meaning they define re-aggregation +differently. Sum points combine using addition, while Gauge points combine into +histograms. + +# Single-Writer + +Pending + +# Temporarily + +Pending + +# Resources + +Pending + +# Temporal Alignment + +Pending + +# External Labels + +Pending \ No newline at end of file diff --git a/specification/overview.md b/specification/overview.md index a542c2b04b9..01c91193172 100644 --- a/specification/overview.md +++ b/specification/overview.md @@ -264,9 +264,12 @@ supports both - push and pull model of setting the `Metric` value. ### Metrics data model and SDK -Metrics data model is defined in SDK and is based on +Metrics data model is [specified here](metrics/datamodel.md) and is based on [metrics.proto](https://github.com/open-telemetry/opentelemetry-proto/blob/master/opentelemetry/proto/metrics/v1/metrics.proto). -This data model is used by all the OpenTelemetry exporters as an input. +This data model defines three semantics: An Event model used by the API, an +in-flight data model used by the SDK and OTLP, and a TimeSeries model which +denotes how exporters should interpret the in-flight model. + Different exporters have different capabilities (e.g. which data types are supported) and different constraints (e.g. which characters are allowed in label keys). Metrics is intended to be a superset of what's possible, not a lowest @@ -279,6 +282,9 @@ validation and sanitization of the Metrics data. Instead, pass the data to the backend, rely on the backend to perform validation, and pass back any errors from the backend. +See [Metrics Data Model Specification](metrics/datamodel.md) for more +information. + ## Log Signal ### Data model From d38c7b94d308de875dd68eaede8bec94610cd055 Mon Sep 17 00:00:00 2001 From: Josh Suereth Date: Mon, 8 Mar 2021 09:38:24 -0500 Subject: [PATCH 02/14] Fix style nits. --- specification/metrics/datamodel.md | 59 +++++++++++++++--------------- 1 file changed, 29 insertions(+), 30 deletions(-) diff --git a/specification/metrics/datamodel.md b/specification/metrics/datamodel.md index 2f9f1255fd2..67f1a708b6f 100644 --- a/specification/metrics/datamodel.md +++ b/specification/metrics/datamodel.md @@ -1,6 +1,6 @@ -# Introduction +# Metrics Data Model -TODO(jsuereth): Write this +TODO(jsuereth): Write introduction ## Events → Data → Timeseries @@ -11,7 +11,7 @@ containing a higher-level model, about Metrics APIs and discrete input values, and a lower-level model, defining the Timeseries and discrete output values. The relationship between models is displayed in the diagram below. -TODO - Diagram +TODO(jsuereth) - Diagram This protocol was designed to meet the requirements of the OpenCensus Metrics system, particularly to meet its concept of Metrics Views. Views are @@ -57,36 +57,36 @@ TODO(jsuereth): Flesh this out 1. OTel SDK exports 10 second resolution to a single OTel collector, using cumulative temporality for a stateful client, stateless server: - - Collector passes-through original data to an OTLP destination - - Collector re-aggregates into longer intervals without changing dimensions - - Collector re-aggregates into several distinct views, each with a subset of - the available dimensions, outputs to the same destination + - Collector passes-through original data to an OTLP destination + - Collector re-aggregates into longer intervals without changing dimensions + - Collector re-aggregates into several distinct views, each with a subset of + the available dimensions, outputs to the same destination 2. OTel SDK exports 10 second resolution to a single OTel collector, using delta temporality for a stateless client, stateful server: - - Collector re-aggregates into 60 second resolution - - Collector converts delta to cumulative temporality + - Collector re-aggregates into 60 second resolution + - Collector converts delta to cumulative temporality 3. A number of OTel SDKs running locally each exports 10 second resolution, each reports to a single (local) OTel collector. - - Collector re-aggregates into 60 second resolution - - Collector re-aggregates to eliminate the identity of individual SDKs (e.g., - distinct `service.instance.id` values) - - Collector outputs to an OTLP destination + - Collector re-aggregates into 60 second resolution + - Collector re-aggregates to eliminate the identity of individual SDKs (e.g., + distinct `service.instance.id` values) + - Collector outputs to an OTLP destination 4. Pool of OTel collectors receive OTLP and export Prometheus Remote Write - - Collector joins service discovery with metric resources - - Collector computes “up”, staleness marker - - Collector applies a distinct external label + - Collector joins service discovery with metric resources + - Collector computes “up”, staleness marker + - Collector applies a distinct external label 5. OTel collector receives Statsd and exports OTLP - - With delta temporality: stateless collector - - With cumulative temporality: stateful collector + - With delta temporality: stateless collector + - With cumulative temporality: stateful collector These are considered the "core" use-cases used to analyze tradeoffs and design decisions within the metrics data model. -# Model Details +## Model Details TODO(jsuereth): Recap split of 3 pieces -## Event Model +### Event Model This specification uses as its foundation a [Metrics API consisting of 6 model instruments](api.md), each having distinct @@ -114,8 +114,7 @@ observed in real time or on demand (for the synchronous and asynchronous cases, respectively). The instruments and model use-cases will be described in greater detail as we link this model with the other two. - -## Timeseries Model +### Timeseries Model In this low-level metrics data model, a Timeseries is defined by an entity consisting of several metadata properties: @@ -140,7 +139,7 @@ is no delta temporality in the timeseries model. To precisely define presence and absence of data requires further development of the correspondence between these models. -## OpenTelemetry data model +### OpenTelemetry data model The OpenTelemetry data model for metrics includes four basic point kinds, all of which satisfy the requirements above, meaning they define a decomposable @@ -156,29 +155,29 @@ Comparing the OpenTelemetry and Timeseries data models, OTLP carries an additional kind of point. Whereas an OTLP Monotonic Sum point translates into a Timeseries Counter point, and an OTLP Histogram point translates into a Timeseries Histogram point, there are two OTLP data points that become Gauges -in the Timeseries model: the OTLP Non-Monotonic Sum point and OTLP Gauge point. +in the Timeseries model: the OTLP Non-Monotonic Sum point and OTLP Gauge point. The two points that become Gauges in the Timeseries model are distinguished by their built in aggregate function, meaning they define re-aggregation differently. Sum points combine using addition, while Gauge points combine into histograms. -# Single-Writer +## Single-Writer Pending -# Temporarily +## Temporarily Pending -# Resources +## Resources Pending -# Temporal Alignment +## Temporal Alignment Pending -# External Labels +## External Labels -Pending \ No newline at end of file +Pending From 12f973405cdcb2f1bdd05022297aafdc1d2a3088 Mon Sep 17 00:00:00 2001 From: Josh Suereth Date: Mon, 8 Mar 2021 09:58:15 -0500 Subject: [PATCH 03/14] Fix footnote import. --- specification/metrics/datamodel.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/specification/metrics/datamodel.md b/specification/metrics/datamodel.md index 67f1a708b6f..71f82d0372e 100644 --- a/specification/metrics/datamodel.md +++ b/specification/metrics/datamodel.md @@ -144,7 +144,9 @@ these models. The OpenTelemetry data model for metrics includes four basic point kinds, all of which satisfy the requirements above, meaning they define a decomposable aggregate function (also known as a “natural merge” function) for points of the -same kind. The basic point kinds are: +same kind. [^otlppointkindcaveat] + +The basic point kinds are: 1. Monotonic Sum 2. Non-Monotonic Sum @@ -181,3 +183,5 @@ Pending ## External Labels Pending + +[^otlppointkindcaveat]: OTLP supports data point kinds that do not satisfy these conditions; they are well-defined but do not support standard metric data transformations. From d73341ba4e708e70466c0e11dce2fb96e568de3d Mon Sep 17 00:00:00 2001 From: Josh Suereth Date: Mon, 8 Mar 2021 12:18:41 -0500 Subject: [PATCH 04/14] Fix footnotes. --- specification/metrics/datamodel.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/specification/metrics/datamodel.md b/specification/metrics/datamodel.md index 71f82d0372e..4d4c419d4f0 100644 --- a/specification/metrics/datamodel.md +++ b/specification/metrics/datamodel.md @@ -144,7 +144,7 @@ these models. The OpenTelemetry data model for metrics includes four basic point kinds, all of which satisfy the requirements above, meaning they define a decomposable aggregate function (also known as a “natural merge” function) for points of the -same kind. [^otlppointkindcaveat] +same kind. [1](#otlpdatapointfn) The basic point kinds are: @@ -184,4 +184,8 @@ Pending Pending -[^otlppointkindcaveat]: OTLP supports data point kinds that do not satisfy these conditions; they are well-defined but do not support standard metric data transformations. +## Footnotes + +1: OTLP supports data point kinds that do not +satisfy these conditions; they are well-defined but do not support standard +metric data transformations. From c0e132629e307b112d5927632e14c12c9c473985 Mon Sep 17 00:00:00 2001 From: Josh Suereth Date: Mon, 8 Mar 2021 12:52:49 -0500 Subject: [PATCH 05/14] Fixes from merge w/ @jmacd --- specification/metrics/datamodel.md | 41 +++++++++++++++++++++++++++--- 1 file changed, 38 insertions(+), 3 deletions(-) diff --git a/specification/metrics/datamodel.md b/specification/metrics/datamodel.md index 4d4c419d4f0..a4f97079347 100644 --- a/specification/metrics/datamodel.md +++ b/specification/metrics/datamodel.md @@ -1,6 +1,38 @@ # Metrics Data Model -TODO(jsuereth): Write introduction +**Status**: [Experimental](../document-status.md) + + + + + + + +## Overview + +The OpenTelemetry data model for metrics consists of a protocol specification +and semantic conventions for delivery of pre-aggregated metric timeseries data. +The data model is designed for importing data from existing systems and +exporting data into existing systems, as well as to support internal +OpenTelemetry use-cases for generating Metrics from streams of Spans or Logs. + +Popular existing metrics data formats can be unambiguously translated into the +OpenTelemetry data model for metrics, without loss of semantics or fidelity. +Translation from the Prometheus and Statsd exposition formats is explicitly +specified. + +The data model specifies a number of semantics-preserving data transformations +for use on the collection path, supporting flexible system configuration. The +model supports reliability and statelessness controls, through the choice of +cumulative and delta transport. The model supports cost controls, through +spatial and temporal reaggregation. + +The OpenTelemetry collector is designed to accept metrics data in a number of +formats, transport data using the OpenTelemetry data model, and then export into +existing systems. The data model can be unambiguously translated into the +Prometheus Remote Write protocol without loss of features or semantics, through +well-defined translations of the data, including the ability to automatically +remove attributes and lower histogram resolution. ## Events → Data → Timeseries @@ -53,7 +85,9 @@ inside the SDK or by an external collector. ### Example Use-cases -TODO(jsuereth): Flesh this out +The metric data model is designed around a series of "core" use cases. While +this list is not exhaustive, it is meant to be representative of the scope and +breadth of OTel metrics usage. 1. OTel SDK exports 10 second resolution to a single OTel collector, using cumulative temporality for a stateful client, stateless server: @@ -78,6 +112,7 @@ TODO(jsuereth): Flesh this out 5. OTel collector receives Statsd and exports OTLP - With delta temporality: stateless collector - With cumulative temporality: stateful collector +6. OTel SDK exports directly to 3P backend These are considered the "core" use-cases used to analyze tradeoffs and design decisions within the metrics data model. @@ -186,6 +221,6 @@ Pending ## Footnotes -1: OTLP supports data point kinds that do not +[1]: OTLP supports data point kinds that do not satisfy these conditions; they are well-defined but do not support standard metric data transformations. From f3587726a3279922c9445dd545e9806f53abd8f5 Mon Sep 17 00:00:00 2001 From: Josh Suereth Date: Mon, 8 Mar 2021 13:28:20 -0500 Subject: [PATCH 06/14] Add diagram. --- specification/metrics/datamodel.md | 2 +- specification/metrics/img/model-layers.png | Bin 0 -> 24407 bytes 2 files changed, 1 insertion(+), 1 deletion(-) create mode 100644 specification/metrics/img/model-layers.png diff --git a/specification/metrics/datamodel.md b/specification/metrics/datamodel.md index a4f97079347..70de49dd800 100644 --- a/specification/metrics/datamodel.md +++ b/specification/metrics/datamodel.md @@ -43,7 +43,7 @@ containing a higher-level model, about Metrics APIs and discrete input values, and a lower-level model, defining the Timeseries and discrete output values. The relationship between models is displayed in the diagram below. -TODO(jsuereth) - Diagram +![Events → Data → Timeseries Diagram](img/model-layers.png) This protocol was designed to meet the requirements of the OpenCensus Metrics system, particularly to meet its concept of Metrics Views. Views are diff --git a/specification/metrics/img/model-layers.png b/specification/metrics/img/model-layers.png new file mode 100644 index 0000000000000000000000000000000000000000..1ca9ea0b30a8edd5df752ae37f130694e43e0a2c GIT binary patch literal 24407 zcmeFZbyU>R`z|V7A}FP#3Jjr?bPXlo2t$Y>-CaX>4T2~V$|&6+CEc9@BGS#!0z(cx zG~CblJ->6#{rBE`*1hYl^AF2~pZUz*?|%2*&-*;j%SR10WpYw_(p$G~kwacQe|75? zF7R)h6JkQ(H#6*B6t`{}Gee%sz4kWRP9;cSQk;1IBs)p_khYtMG#m$e3~W@0 zC=ovjrIi1F;m3#im|Zi)ST=~`^C`AR&lw*3-Xx2%Lt7wFF9FJ1q-Gt>;dncZ1!7Uclv%TIgxwM>CX0a;a~HosPfVWA&ISC-Sph|3Gy| zWhCCv%oBPX zkDe#DJ;^`JDrrS+{**0l+@CSh)6PM}tl`8hoE;pmuN>wook+V`yCXYAmHb$w#3gQi zS0aw-`o;Jf77N{)0t{RZzYhln%}5q72QAh&X6G3N2~GF7?Mfx)AT7^dlwEEv7`=#N z2|8|HpRP^+5G}Q@-s|G*T5{-rIUXL{1Qq`gM@VXI?CJYXm~l)Z#jh*zc%uB~U5t1J zM#}nn^~`r;?6-5;VC%(e*MV?VasJ*~T-Niliw-g3oTm2?-+H z+(hR3*y`*;3q6_vE}ytF&W!ek6|Jj-lyL$ zvP?U}L0f*u8{ec@0^RQ5+lVYAr#HXSzxe5KygttA12XR}{#?{YNGduj$vUmBsj2D8 zx4tExw&S%>GVjBsk*|f;CHqfW(pCRCP+=#GsFe}rc_J~kgs;pi^jNnBjvH+f-|Q8M z_TFAwuh<%$Ar#dpYyb6CE2hZMYk9hd9a+&*APe_DN;bhKT#kxA7SnHRy-06<;^X#x zqC@(m$RVTbXq_D)dhy=k@DQXPm}FdWk?G0mhi*EqY?|ZMpPaa@p%8M|Q_ymBE!Zo; zz`1BL*l<*3!F0xxm=-NJQ+tMW2ix>)a_aTNSEI!s+fDEFFuzz2DbX~FvuMeygqZ+t zjeM$;ZeT}J13Qu~46)chX_&5_`a?oF;QUxE5Gjjey|@O%4gY?V@F8e(O+RSVV^bXY z9yhG~Z_gMqT2wz}%FaIK1T+0LNpKUV$NKOuWkIu!d#~c=PWbAh!*N4NSpO`Umlzjp zV*7CKF9jsdG2?UbM_HlLvb4pvlH^ob_}cB-v!#Z{(w8Fz=U#XOA(C;2QE^kAsNz_m zBp#hLaHNhi4&QMS-DqhDP4(O*&;3`4Y{TkLD+y zdYjh<1}oxq{@hQVs4zZmb<8N62W@$uRlqy?DGcIQB;9|@8MPn=Qj>CoX*e~^mz-4I z)G){jDT`?6*p3T)93A)}nLCN>ZXuC5^*{Giy&;V&7R-GRVyX0@PMAt-FeO&Z$uHdj zh&%*ye>#`3(a95+AWl5S;jnW;QcV?CKClrU(p-aNuJ1*bl0VL0%}5Min3nC{46TfY zjsR(}=KCWLndJ+-{;ks+SGO=+8M=rfO@!aR>g)#TD+z+~X9gjlo@ZOxJraO)(>_YnAY*od!zw zbacBNydl%B?*(?OIwOP_0r_N5*^{vrb{s}b?SPU;vp~s?Xfo=9g@8Qnmyc;a$f#XN zX08)~WY@IMi47u11AFEU)L%yalob|k zr-TizRKhx{;T;%#Zy#Wd23V(q1~e@d_hj|-DX1w~Df6M3Tp?=kX>%%mA$LB5{*SWo z!$)OHiNif1^yEnEc5-Trgm;X^8AkbvoWUl2FWlT-87V?rzv)jp#GA7skTke?VLp1K7HnbviJ9FOC}B2G@^rBkAcFGntNMC8ZFr_myB5 z(p794Wp2#;?#HVXX~pp}Km6gim3A|kNz)&zW@X`LRO<%6gr|dVX(r32Q(J9T!QrXi zpB}L8NGQGh9CHkun64A75TXhAJEE~@Us+F!7oG!mfTjKYjR=6K*rlzoiRm_VHvTMp z{XrDs)sY^DPue7Xk8QDT8|-(o^X-pj8!Q^y0_0wmJ9Aj>R~Wa&#ah$&9erQhF}5KJ zxO)n3>T{R4s&N+oqwDH3y?dPeE^l_up4Mb|sba@^$l~mf^w7bWbcK-g1EkVM%~U#@ zemjM9?72nls<@McOEz27Qe33T;{6{b)h{b}C1Y^mgmR=fAe57Js6`cVdscrQuw&}2 zc|vUzuseosU|=AvuFI1uHjv=9%im%_VyKk*Fr#-y#16_QxdjOurBJ zR&}F&XV8h>#2l-+kdY?P|ELDWB+M$x$(Z}W_X|712|$qk>R5c)aib$!L7f3bRFXDGct(+RN#E9p*|#>KLI|KBCU|3?cW zoPzb&`v?eGE>HtHdu-?(H{SifS;DT}i|C0qcJZx6&joHu*6nhNYGP~RL(aaEy6ve) zhve5zM^md=*kXiU%%+WFfAg_uI<{_Jbv%J{R%c8Ie&2;Wiw3^ zcfF9WsFb>4MGHLmQ>r6Yg|}XN0Vm+P`ds3N(qPk>&OJ6AX^BI4zKTcuf}QW=(`_dX zpc?;BpZq(F&iHhmx!L#fWY*StjEu5E0#lWpeT{b!K#St@^m_R3vCntO24ph*Vw_~0 zDpubQ_pwNA5fJGPdG9Rc z+e0T83jMVCme3m5`MTk}3n!l~Z^+#WszG`r+e|3)`LxSwN$jWm&7KeN3%a{oI5t;{ zo%al10MVTXd&@&-;g^AbuO@wQWL2t0?98tv9Zt(W&EjF-mc=wu8?4QGo5B6s*O3wVB?Z^TDkr>sbv^fhmp#8Q)k$w@UeNO{CkSf1=tW9}-zdAEhjA!N z`<+~JgOw8fG_W*96<$O{1UR{BOGKviaCIv!n=e#b-|sSrh=R#I(a+%DBNQ0)c;00C z!d0OMZkY8#UW&t=ph>9gI+t2ym z3j*HL($EBWEsT0yB0UbmMrIgF&K7{cL%sg<228tCv+v2JzWwnV@1^FvET=j9z#W}& zmm)ut|9svi#M}8|uJ)o0`{}T^=?rhYeq8f{)i_QTcM=B5wIhUiCqA4*Nc>OCK4iC4 z7l7x?$=%v+(TL*k0bnQrfrvv4LFQBu|=Kc>GvsseU?+o#ryvAn=FvLxQS5nmk z8wCgl|Sb#iqgCxCCNS6E~`ctzixs{6BN#IwTX1-cOC?2tQ!l`m58}leCb7 zR2Q+O>QyhNjTnD2!BcN}z-lL7A2thZk6MFTev&o44xw6w0s$Ba+bx;wD+ZH&YI-wxD z96R=XqUi?x7^ifa^f(wy*(yR*)|ko$$=0VRTZ*LkJc@Z6W%y(X8IIez8wTX7?GW986if-&PX1Ga^9+S z6t_L3Fa(kS+PYeiuQJI!PuU3MFvntvfVa}-CC2~0)eCF`eE~@ zPoG-qH?Hk(2C-Q}i*&^J4I0hCJSb)d7I~P$V-(h)OsMGWkX+m_7Z7xTvsN{u?b-MW zz6L%?(*q=Zc()CwM%nna?mBh-H;6K1)?uX9t?-ZspESp^ekun_DvKbCPQMyek93~-bXm{cY?8MgZW z9ZkgUPos;yu6*^@d71clI_rfFq=eYb4)i(@&;5RmI?kKrq1PE@rI68ZR2&KQwdJOS zZ(NV>TuyZIKupxIMn;2?Vg;50#ExRP{1fqP0r6$8dz-N60|)Sg$0|mVW{f* za^O@Jcm(H)pIe|#;(e-GZ$}p=eYNrkQ~0Qza@KZ{B!>d-K*#L@^HqDd}2 zNBM4VtlyJ7i!70rO_o**9|%0F>mPeFEK1ACzquHt1GAc#29xUA80XEUiSBf#7`WjQ zV7er5zdnucK7U*}`#Ph|!}3JMI9w*teZkyyt3oY(m{h2MF;6?tBPejXNYD z6}wQIEoF zx_lLptz`FwPpCMU>YBOkF3OxpUr21#NcQy-V{T}#s zACn8ZW%oowkbfZOjzZ4k z+VvHG72cQ(vFES}S+^g7eEVg;fT)u_^j)+c?n#?)Coz;vCDj=EslCWgypu&dOTal z-GA8Hh+wl`zRG2yVDr^6Pc{v3Z4e7In6+hG-*EBr&d*Ah{X-r^qZ4yB#e>-MWbR<` zSX0rmi=w;q866f#NM@+GK=!RiotjoSu*uPgt~`VpHhB8{`mYcF@F>~5^lF)2lsJ?B z08_oqEOL4i@#)hcbchv;4?Nox?TZ>W<4X{HU!8)Ea`UsYSfP|hzyI_0Y&Too4GubE zw?OQ`eHPKJp1|Ac{7eKUJDf)_&!n`0F$n6^SGcXp7+o*w4))T?!-B5$gd9$Xg6Qbz zcGsD8#yX_%f>z!sGb9{Q^15VAaGBfW^C(=@_SOY6yLFwbDJJE*UFF=$0vE6C5ndiIan{1{^b%>llb zCF@X5w?l$`BMETj)(#g`6&-Z~O!-yi)(YA801D*Aq*|(^wBu&b+jatF(dN3G({Q5z zU&~f~{@Tu}{vbAu1g35X*0rjxIz6|zjP7Jq3yld(M(AR4Dr>R{T)(fJ3` zR`p)d(pUL6JQb%MZdppEbcn#miQHQ#)pI#cGPOT}cLm7Cmo8QU4;2-`5kKkJrioz3 zO;G)F)C7rh+9smH84&x~N=MIz*Em4)^3vTznsPu*FXFHce@|4$h zl0=;f|4w_=8b}xi=hy~|2G%g7=nuwo?fAGlU`D45?tcRnlP@vcBX_BTq;gf0@D|f> z-a6S<@vH-^HG^cU5#(yDPH^kgVDjm1QBpKgby`UPPQ|?jz(%mI7nscG^B;HBJ zx}T0tU1US_A+eB7k0{;g1bz>zM&0k6oz0?7h>Xn$On%SuH=TKa*=Ep_N1rZ+zsEz* zH96PH7ZdyuIN?sn%X%Y50Xh606unD?6hm|$!!zRwe`Jo@NFZj$G6fQod=>kh4Cju| z?4X%~JprkEKHDvu{O-`}2L=ps45)$Xd-+#(n~dOo_7tQvC?L%~>FDokgFYL?{1%>T z-&o<)tuL49-7_;eC*fdigYy?Yj0ouNuJ1a>^`w_IYYFO^L(!!v#g= z)56UcnCpw?!P$M8Wa~fNPn0^}Q*@qH(>6h`%{J`_pSrjF<)OCjSB|rhS>Qqh7D7_R zPy_MIE_+Urryk{2s{rY{^s`|lhZdJ3+O6ConR$DX`)n_kz*@3vj6$zsg^XHEJ=(IK ze*%?gS-XM5@uBh$A?w%&%BM_UbmIxEf4RiFQau|Ahf5t9J99nHSj1?GaZN~qn)Ad? z{56pB=VgB+z)oFBiAxSH zP(2pUR!m9?e{7tw7=8Z~0g-_^y5JGuvf;ARrLb4uKAZZJ(_`2T{S}gka39h zW1s{jDyNchIr|oOF@S|*3Vibn$J^t7QHVjw%o^JE2=G0ZTXJt4?d5rJ2-60>#h~t1 z69nV62~buQrJa#0cRkKo3jbD>TL)4SWmy<`41!UBP zs`SpzySA~JXW{hh*P;w$)cN0+lZa^-p4ql_bKWK(#nw9&2L}fyYT`FPjSZCNXBy6H zY=_(0@obEppzCD8aF*4gp1amR<_j9~XmNUEp>h}`jr;GP59)SK-1V}xZ&?A9tVjnM z&)Px~-|El;NRd6wBC= z&};eNGFL;_ky^O5S~|5eUf6fHjh>`)IVK#UR4Z)Kn2`WUv)11uWj&vS6+njpb!0Gl zA4Vb!Dz5O9OjzR7P4~Dy;%nPq9;&zktqB&q%gZ*QoyjifwtlYQ1^~uaj*Tbx5l)C= ziJpx`hdFay00vEHQ?yA{&&-DHPG!nMMaqJNl?yd1b}#qv9I~G5>P20yzn)Zl zxCx)6L=O9qxW}Z9S3(s_cb$%8a)W6!zQfZ6$B_$xuAGIkMemg4Fe@T-CON-NVA%~t zd{R{mArpf*Ea@wDcIJ11EL*M)m(DY?5qO+cR1_(y#rXXnaKlqnR1D`U|FeAKy}KnU zg)({}eR?U4B?VsJkOI&zi%HW!Q&V6oK`Z8R+yfmpRzEuM!S`fG(XZ*>=O3^>;g+~Q zUH?NQMP$sm<1>WToJ=Z-eDe>R`DFvgUxs3w&bo2&IDYxaselZ9&`XPdh>|Ko!lTf8 zb*~`2<|WnhK_lseMJt?>3;;WY+Q=|!&J~|4PxjtuHe;4PX+ms!`9MrHpnqUcDKDCa zsdQ(wbnxT^91xCLzsYxMx|sn=*g}7nO)RzcoWZ*Ncj51ti8+fMmN`CY+Lt}4fk!vX zvY{XCfL9O?8?6s#Tb%&#TPD?i{`Fuz-7Bh^nU(bJbg(X#BXj(!rhsC&+{5x{GxSjM z659L1mke4r3NQYR?$c9Y--6Y1%-&Md|UN74Z#H6Jhe?ZNWvA3{lYHQ~fBm-R12Tyt3qAoK>s;#4ugOTu;eJ@*M z7S4m{%NKDM99wpBWmLtFf32?(m8iTIZisIE-mo%5c1=eFiV;KoBO5&l(;uw~HeG1H ze*I@%Z*Fz4{M}9U+jJwDlB&(A8&$A#?RwLOwIbjs$fLV0l`39I0U+J%Yf zJnXWsnMw5M)u^|E!y^rB;EhD&5B4+xMa#I+qx5X|_m8M3&lD*xiJcBamLy9ndm-BB z>Ki0k0vmgQ7(tVtA%`$Q-)+}XI$i(FTEaW@d!?}>vAfV2K5Y1~@Be@PhZ*LKSDOtI zJEr|wPy75Gpv67NI%)OMie)S4CvIG!=YI5DtVtg0QF=ASKB)nCl$K~-0Uo8#9lZ7L z0guw1($83r618Wq@N4-G&fa0K0gqBnnMz3}V2kQ}k^|VH9)%gYl7FLo%xLroYm4IA zj@EsFGif93z`z)Iw~?-;&S#JXsF0UtqARov!4 z4-A4CXOuoZ&z{F*dQ2bFv7<2Ed4gNTL8!jT<|dwTR=x!mtT_d14oa}e>qRQRT@dN- zJGpFB_c`FdKX-3D5%Vn z>Ac1kF6^1t>-l!pzaUdL{EoJnkI~2<81U2j?R&%K zrBXV{H}iSBd+0X`Nv@k;dPNjaEcZML(92B7@iD>#X%C|wZkCx-U`DLoB6nL-y#^0#YXiJW`n^sNwLTk_KaM$y7(CX8+VvN{ z63>d!p~I6!IKDR0$$ti0arX=KNFo*vkoZ9VV*RQCphD)l+B8)m12 zVD^EVXtNl#ljf*L&$$>{d12a$GIFsJvia4Z$i>bK3-8PUA{TH#YG%V7nd)%VJTvsW~sm( zfm`}{fQ#mt#;?4Hi0EO!1$Qt}OLR3XyL9K=77zd=jX%+(fLPeFsdPek*y&r(hu73*5U#~-re6{TQ zWlj5!wbgOqfNavy=QfHr3G#_!X-vTmM5G%+JS?(Dx{>m#Mb>mVr#~C(slZZyHXFk# zN`N;BYd%v%?o@cZvR_1J3kcJ4E;3TcTz|G{mDyneM7nP)otmFL`%~X4{V0|OZz5aO zT?3nToSM4c4^x#&8#GEg9sy#T>OpSHVVWcxER-gdI}7W*O9Ct{$I-{9KPVI#99!hZ zvfsw!_%d48^jtS8Re}TSMF#J3Fve}i)GR%yHNJ~AM)j5x-QG+u3c_mnV027B%1BGQ zucXZVmHB6Bl>c&ri;3q=$r^x3E7=3e+3Ai)ijqjFhXAD}d9pp#r?+5ued_MvZQzE? zKBK)KC+6bEV*9DKnnOwjUGA}YH208iFe(Y2m!m(5%VZutoRX=`%i z^2`1GUjye92d|%F3fWWx5+Wq`00Jbo(6Ldv{TAvcn2>@Y<-X53o=J{COVtTtw|~$G zf*S=;FzigbYEr1v6^SX&xv0sKIpdlb+&3Gq(<@aygijW`PSVv5T!I|HCrL%l_y%tk zCevu(_b#$M+o1|>+!}RD^A8~BdJIKN-*tsKG}@;3i=59r=N@U@QLWL(T7|Ne{%RX@ z>DB`NA~ql5)0%k~vkIrMZ%P~|{%gi*>KIXrmUc+nZ^7KZY_?MBjQrI&XmJZmJOYXB z3YOY4_(jBF9@j67)-S*<2WwEPh(44?ADQ1HGB(so#jli zWz{_ayQMGHSM0L`k>tYCL+QbuXWi&dpioVAKqe?uSHK#?xJAinBs82aI3EtcWqEv< zMUA!{?Z;DOaXfa1c{cpoVT>$l-`d)n-vcr)r%R5fxbZ07K%z&UvA1ipTzvG-BcE@aJvS-jEcDvaGmSFB~pr!Drjfr*CUOZlNB za_W3V^K|^8>42GBZo3*C?Tm+^zm{o6#J>Z4^ZYzknD+G%UGS8To%C3YD!{Vc;$G4_ zvrZn;C^K>}VXB9ShI2UKGQN(w7D=Bc8_t_eu`sWegjFUgz%OJI?Ja4gAHpEoPcuHg z^8M^5R3YpRHT^x|htQ}!8>fu6^=Lp1wTmx^a=9SX7*!z0i$<38(Q)o=lQeB7*U)sj z^di5q_7jqxNO|}jNFpy~weGcpTIXa@PtN!ev6j}JAOnj!4x1Q2ahQ{4mm63waNsNUv({%mRBI+%&ICrH+@Vk<)BA)OW z!#-?8BRG6epWldRm?Q?`90e_k%CT0zoyCirlZAt6#@^RmJP(pUFS7An*$w~>#ud?Q zm7u;f0OzN)^mw09qq-KMw&CCiA?{nEKa&{G(q0K(ETW-&*Z&DCm8+t#98#lAw`^ z7xw;~Q%Jd=gnAq4AwmJ$DeB}Bth$F%dn37X4SdZQNTYbr7x(sg|B!zNzIu_pZHDXu z4zO0ExN0envT^`(R)CxHHX3~Q%KDc|&*Uhd1p8UPaQzu^G@xw-*btLewCictyjO*X zP(J>Iy?kf%GWJF;2H9Y59R7;9=)8kPd8^`=#THW@T}r z3~Zji(jThMwRc3x0ah^8$6XbUNoC8^BZI*=khlWjW!=%GN%9vyBGa-sJ%ix6kHzbL z3eX;)9D>IEk3CXMK#Grl-=xqw@m^B&hpG!#wWDa%>D#pwCxFdieAZ2`ptgharmLz~ zFz24KDeBm(G-m(m^(R64U~W@MKQp)Y%6gjQgU5@U>kP7$q|I~Uq5~hVglIkpacE@F zb_XhV&>Yf#F8vJ0Jz?{^J#*>UijzcAp_aICo4~>ee}EwTMeDsJt8Ky62VdxVem7^u z+&)P;#?DC!)lj3Upi zlW}V44sFj%njs`4<`A@+B_+qG{EFcAi;gdims-}XPhGC4fc=R)v=%oGZ(NJ;4Dh(DxRvTklyVC5em6`s8|?VSBeI@2>_P(K&4smt}CNq>*mAyNpPpl z#%)B{J;xzN8Oa$lqkDsu;@M9QNWxR48vksExxEtN6FND?f9=GQ>~D5Wfl1`6Bjx$u zmFNHO>g&^ioUm5_Spm%l#5^LbEc$ zw*S&acO94~U3=49QF!p(+mJ{U@sIA|`EAC8EH^)Ejgh>{Pd~f8-)P*-%v1L(KZx!d z?au0d%ydni>mwB-6iP9M7$mkXd;Frz7u%x()JHPiKJV)o0OJY3Bj-MwmsOfuiB{x|f8X4VUvPGLS4~`k_Br=N;tRCRW&RuW9S}7gT|o8H8iT*cR(5=oNe0)B z+j?4S7f!bpS`Faeh=*G?7`AGX(89*^&V!LRbvUwa$3Wu>wm;+A*w%Lc4jKm=>0$76 zeO=x9E)wW|=z#@b3Q?E(h5wi>!1D-05VBy0ZIgFOopt8-w`a!3dgQ7JlKb7)HM|qyZ*4MOFd1Dt{1NAGm^ZXsLtKG zLc{;f3w*!5mVw!CRqmhK9f)};WTJ6%1B64}`9F?1PgzaP;ml9G^}kO3$?8}J6;Q}q zmlZs`{_86Q(xYmOLmkMWGmgm%b`5VAR+)9MS-0~p4DizdmE1oI%#DTB7)*8{9T4nA z|L2!$s_-=aT);X44Bb%QG*N{EH<1)?c9`|CuW(F{1yq?@b*YO}ZTpZZyKtVst@%o1 zF&d7#Cy`hm#A#D7tlSEuOPWPTm+ zo1V@-UB|e~_ldSHFN+-B89K^CRVlvboNL0^?08@`OvNl9RhKXXzvWM_mr(b(x}7lb zy?mT=o{0zL+M*lf6r#3evizgNR!)Qlhw+J)7#C-&vzUqm$nIgRQwXrM9h$U<^R{$p zvvyEQs+#$gT4w3SxkctWrUMH#-lp;8GLb$Wq^#-S45P_^mn9=!m|wkzh0R^KOMhC zI0d~af+xD`@K5xEpn0G*dBl9JB2=%lJs?ng0$zW)3X0n+uoX38#M+F#5G_2lZ;1(=RjV3DucIq}D`S-*kr&21 zfQ1nV%cU9pd@b3<`CN`#+|_XSa;45-QkCH2A^d|V2-6d}pFd~MpC0{KOvx47VlZv4 zmfbIUTCBXw+~iSXUGr=M$vr^0za}F+ahWIuL`+ELvlElZolk*`g9&4au{FxP#azx+ zVgpb663#CGsP`bd{GsC`gj40GcKb= zG5)1OOx)9%5Va-8g-0Qe_(8SX0}C$SA$a9xT6CN#sr^85Wcfs*|2a2WZmO`(jrf~O zgRy4i{y`>5Pev4weIcFGy*UBo*L)K(xg`aF`>9=`3hLH}Px|yw1nP0U1KK&7dR;94 zrs$??W)8}~fs`0OlFitAZurV5{{Y39&t-e(^|bL$iIA<<&mwGkg(*WNlFVaSm6W7n z`#J+l{#*@&GjKA}ZlQG8tHodC?>uCI-WJgqI(Y#c1D>4R+!Aizgtf1`u?n=J;J_!~ zSk0tw>X7zi2%o?1OS~K(6PxB!)!QKlq9>Uo6tF-042}!${yghSfk5vXj(YRldULtW zVT@9+qtDkS!X@x?B8R(KhK7;z9tg~jr5-ASiYL4>X%}Jj3sD55rAbwZ%llFdLh&e8 z#Lay`l@{yG5x&PO!q4Eii5n)oxr)6Q9Jm2WN~Q_u&BQJ#z}Z|Ed!?tjo7t=HdB(gm z{a^uh67q0+Bt*lxCjynhE$nD#I=*F}R8?crPzfe$Crf=y;XE-q9c!}vXVw)k;MGm% zpt>wx@g8|)JP39pI=7vD6>mVcaGFtyi_Js>vy*ROEo8LlJW0LxYf;Ur^eKR>8VG7; zL7yQ_i}6W-B6l57Xg*9^dhuZf8b=3Fr(@QjQA6&$gN_Y>6g(EiseN=BD?Wvc@Z0ox#4yN9_6?)bR< z?4=0}Yq7wwR}z&(B=PPZC4T|c$)~GnO{zK7&BX;#+Z#Mwf%A7MDJr`RTdklSKGSgJ z*zPl9tC5R#w=9U|cY_980>D~J`HkI@J>AIjO0l&LaANA2&y5Fejn>k^3U^3SCa*yO zTOi^LJAZ4f8}gLE+hUNnO&bl>cV(>`--Dj%P30h+mO&OKq~?rla%2N@JNZ2?qTVN| z9<&Rne>x6^lzs^$>Kp6((&4duYU`BoAjF5ZB0#-6QG)1>*aeMXA3m9bTQVg1u^RvN ziQs*jc4CqcCJCR0D}g#~ToH!gw)cPn?-Nv{1-N`^ZoO+{u*?iO1f*z8?qQ5GtiPDmd z^BtfW_z19h9<3B9pLVR2T3TA(9<;xM0FH}i7^8m^fCCe-=WuB5rNiFpxoP6;o8yff z2&(CgsKeocii(5t2sN74qn(BD-uKU$rW5ZVasR;E3~oo+vLXg9yONcSC0mYQe~KgcDQ%Tnu+BZGnik1Rhqt#hQntF~8r{ zkeFFuGVlnGk&SL|Q0W_|FIupD)%4ZBejQf&tht=>ztJb7Hpu5O!@K~Z;U>r=;a=yn zd?z^w4R|Ow611WLk$$T`nSU zohW$HnXgK}Kq1zhv+%HNFZ1!lK9sPf?H3F;BO5BQ`YhdJt?#Su={f5t@1d}At|oYR zYyqW#K!?X@3V1wgA6e0XB_2@?=6DRQ&OM!YQ_d{1!I7N2dDEVcfPhkQ6YW<@B?E*e ze~PjvJbsY35WA7dKuVoJ==W|GmYcIie|n#Kav)&YF*Qa>pi6M{nM|C1&m>#rQ^yg% z-}P?Bb-TKR$?lDf=5rP!F{<^fj@lnR>LKrVTiOy+(TQB&cLY}?N`yAR(-d0D`lqd{ z=0=Qv!3Z#{re7XPnm)EBmQs1e48Y<)G)F*-_cbPts1)`M^x>o>kQFP+1hPDH*z>p| z#9U1Rs^9(o^V3ydp9YW8rLA}Viq2}~QIJ*x?;j9k_8I0Di?5z|e6--rM9%nnTkUP< z`N8C|2m5D#IVY6M(E?-+ z<#Cs@3wKPmXOwmmm+$Sv>55vS-JFA z+y4pvgz!cHtHz=M@#$&XIDNf4P?0@Jn98<3;ak8q3k}oBQ?$ApV==!)G=b<&*hyLh zu-lgL*Qno=U)#RQ`xX|Xu=Mn4)0^cNr`~s?nTbT%;;rxqz~sta0~vLFyo zeS#U+?M*<)Nz+$r;PM!i$VEcZNVc|pO@f%i<)g~{|p(TW^>4hY<#*Rn<- zO)d`vYKaD8=|4WImqI3;1N{L>mc)A9;#nji8Dzx%Q>by!6)q@!2ixxx+X9?e3v>JH z{lz2r!=IxQd~JlBHG&xbri6&9Gd@?)YvHwDR&~B?>a|*E0=nHjf=%90g2PqGyZVk1 zR7F(TNk|FBb50AuS~zd!q9?@uIlXYtUiEx!mQt)Z5hNW{*nI|FG{Ym90!)m#=oy!$ z6fL*YAt4!!Ul_hJVPS*Tut1z5Jij3NSZNtMn)_VllcQ{H!ku!7$7esUm}4*c1^Z&j zscn>;YpW|dea>3NL?*23kMa5aUnVcA0$vjd(DCNn;UQM>5I>q2$)yrKhxz2W&NHK^ zlat3XC_JeSg0-$HAwRledW&k|A^OKLh)P%yUGfq6eAqSNN`a6EJY?%N;A?zkF`||L zAQr=PFzwkx3Y-r1@;H9m9RexaO(N1bpatUx;6^N7*eQySp@1IZg*o)}<>C@Z$R8ld z)96F4*~UnRcH#E9HE=ybw2L|HL_yUzML6{s;5%Dq5o_|Y&)s_7f_7rCj!72mfgFVD zMFy=&0D}f}vA+q<4sH=}0DI;B79(T2o(C|p?vax5{A zayNk1_Q$6>`MlzAlRiv%MkO0mL6>;twDn)=;9yf{;cCpWcpiTr{kBvYxL(Cfwo*{k zM=TlrEa4IZIdXV?HIq_MqOr$QyKnYDH(w$4bYgEAIGxcw3GOUM5!Ta~KRq9t?`@eQ z?tC%m&*>*vSrQeZ<}|Q*m8bSi^%kS`NFqDQW%S^83IQk)X{2YSRj~c{8k;6ac)sg`1IL??7a04&Z~kMP zQ?fvXx%Sc%jcnb`+x0(d0GwI{MCq*@g+SXwl;58CE!ePM@)*}Lem*1vk+~BdoZUh4 zW+e{Jz0(7oAsGSwLe>59eDm}@=`)bO}zH_As5EM^Bz@1-3c(_DK%*3kB zNBB%yP(AX2KrtBO>l%@^F!$tidXo2yBWDLVS|_0B(Ohk~ln`R*^{`HB_*1~>3MYk4 z&Z=Z^r)@Fl#i!s?6jgST!Q0+@=wkS7GrH=VZAa+e$Eu#|AfhE6G2IO%@5?!SS`fB0 zJ>#d8VY&MNPTFH2QtR+Y=g-8vbJ`I_`QYa@h5=DB`0Z-R9JfJ|J>q9rQtrD#_?;MR zCwTD;z6-hv+kk0jKs3x;21q4;fzbSv>ZMG2hz>%R0CY`)f16Wl{I_rWxwG>cSY|=~ zrz$gl-A&ICyK}1lqpFUV*$>RD!mFw|^0jl5rc;NoKr$h3o$~PKd^q~70Kl1!~wu%%MZU3zE{)Y!y8?+~18v~E6#ZTnzI{PQBU(t7`lL0TuiW02K%33Z5{>Z|mEFySoOkd%U zoaHxBL_*@fR$RwKfVATK6cLg25g`%-Sc*f}udR0v82vKeXlr*<{~zs~`8Sk*|Hoyi zC`+lRv8E^`B?eimv4=_#*(yteLCH2VmM9Z7h(4CFjD0H<8QTcuL&Dg%88gV9ZL*9Q zlka=_oO6Hwf$#U6``qWAALhsFI@k3+*L!)r9?wUU)`-uaO)n+;!A>K~${Jm$*C|om z#aYUj9cGOJ9wS7u{9Dwus6%$j6U152&S#{u7JftyN<;%+&6R7^KWSLb?mE}b^c1XE zMZ;g=7=hr*tgHz|un=uJr(p-81OUC^)b|;JC#{8rPR|0ckjWl9!J0jXz$3z#E>4x6 zR6lX0(XqQI#*d$HIDF3Qivek;7ph_EaU=fkbExGek;CS`{(tuF)%)_4Bd$j&f|H#L4 zopW`XLXHDokC2sW2kb!y{tNg0MbGKnUyN-D z8@9S?^LY4VR-ItF*dG`Vv9Hj-9;sFHW1y{W-4y<%V_<`%j!%G;dpx1YsGh7huce>x z((wuRM2?Xsw@AdvTZ+XMO*_x)TILaD(TVAYJ44aJa)_nejuKE;Z*;;Nq4;Vj;HeQ@40?-6lu>;B`7At=7Fw0+FV0O438Y;rFUa|Lf>)EO$+peB z?WhDcAJkm=nXr3|>P~4C5^fq9`W)fFr0Xdb7oYO){HOm&(4($56-WM51BnD9sq=sb zCThO0*|P%r?RP*qgQ&d5wtFeBqy`Po*&{RANxEUaL$+xlQ^T#*zvrsL>y!)W0JF=G z=7R9w^c(d`EC8u?zC$*rNQ$m_a1V~sSb`p+avpjEBOso|ZWh`27T?YSmHx;Nr0K?h zCE{Ro?v{d0YolRn4_hJlfGFK)H`_?NZ zVN#uyWswDxcJ=2L+Xl1Q=-E1?OBEN=gl=7ybJdE=xl!D%G&8$YE)7UH;>zGmJWGq! zoZYIVa^kEGb6}$TF23#`&?C@*4us{~d*5FJs2h?AyH$@@WxaEvu#3-i zo5N@9C7z64^w-Z%4Y?Ax+{`S+0+@$iQSf`sqE<)x%flT;eG+u$3glcAn=F}~($PuW zmh{%z_Nt^*gzb}E!}n`>BeR(;wzQoB(sNWGr2 zC%=*r;XYump>la!qJsVvX_gks=lX+Yv6UsXcL)ufa$a$+qgOYIv?|&7h*vKc)4{F! zAkVEt%57^hLs)h`Ny_V~%_rKPuV z^$nA4KQ1ixld*57DIRDNsgjhH6-s%qg2_pYS zDupdnqm+dV{IqnBaZkJ0C`j=H4aN#)d4+Dg;E+P~#-!Fp)WvL2;9CzV=X*>25uxk^ zG*RKZhY|ff+I-kTkeuuOdob0Zmj8B-S}edYRvDN^7YIbOt2*}BbC26fPxtF@1H`+~ zPE`)jRR4Gw>(DK!k6~r$l}igqF);zMeH$o$;kgApdppHrsHqTri>p8b^J`jcyxt% zZ;!lX!?_fiJml2-YYmBrV8=~K!NHM0=eX!_AT(gsYiu@9iL6gQ=h~fg*1p^e1az+{r+})Yiu%Hr-F(VCA5ynV z)d$;mpm2gk38k=+Zljvi{Esb54f7to04s zG-Cg_+E6+HBw4|z4alUFsz2rQBK94uQc*XZ15#1bXmeq%+mm%$YsL%%2W|#6wKW64 zilx67kN}5^jtv%B(mr;u5z-;e{a6dRZb--%e3sG)Ks=!e17#h0JDfI#wq6ENOs#vH z8QF{th6$Gc^2_4CGMmGL1w6I&-QEjAP7S%pxIG*pmark+KVM~30d#6xc9Bq=7*Q~t zTP|I20svHgxphqmP>r2{s0LAp#ohorv9+x$k&*VT_?5aec|!B_O}n zoc60QGlTaTt7|+qS-d=O!N>DqiNG7h>lxu^#P`7OODEl*+&2$EPA05_FCJ0NoU;9D zGRmExz6VDe$-uaXuMF!pM!y^K#itR2dYtqZ6ZnyA6z`u$O3F)GZBxyoxfSyd7_Qhp zUgT*lEB@wUn4F@@-fx6)kuj|gnBp0^28zkb{^z-J>xQT3Z9K)NsZsI6 z+Wl5-5QbduYPza$2TH$}zB|kUSGO?B_phZ*>XmH&&XSO!6(KxRT)$<1w4bK7xG>dz z$`W)@Uw~bXs96g`v?b6&rU5bm!k>qW35;K4f!O_e<&^Jh=YOqv&WQR|o%k+>(59qE zPvjg4^Yw9)cs>l6XPe4$O92U}8a!f5v8_GDaap}(-2SJ2`-VW084j(9_mJe!xTcIU zf9qZpznIkadD`6*d=mO&I~b;Mi*Cf@m9y}j%Is4@8>t9h`PuU(y}LGNUAu?;wZZd$ zHOocJH0c7oHWeBqc&C$ojdK?X>$0cvXtTiPl>OsUUf58C@9_BR?fSUpo#Tz7**@|b zR0jwd;&|aq!cQ$gG_%5`=c%MHZh?3HuKYPiFx`CULy;tu=Zrb5!Qi^% z@?j9${?vPK`Scg&kfe;r$b7zf1Y+Q}tP{k!ASP(tT^;+0H&xP9 zO4*@W#@=>(TRktx#j^cT!`6b&^rtDNL-y$?oDAm$((Al^6x;ErnVR)z;dsZF-AA__ z%FX=)&hOfvt+&9M-iet`^N>h~P@ulp|3=vU-Gh2tJAOY3xE-ZlWID3kwGNyd87&Dt zbcxISN}L1EfM@O9YvY^c+dZ1{8Lrq{ak#0(=^MMW+;igk&eki(@XcDw{FMVbUV*oC qI|G+aot!%}l?Zh9{~y2K3p}r$39p&1zc|dkypf(6tmKMg*#7`b6r@W4 literal 0 HcmV?d00001 From 7bff96b8c8a085c9f946d5a90335ab862ef6974f Mon Sep 17 00:00:00 2001 From: Josh Suereth Date: Mon, 8 Mar 2021 13:36:21 -0500 Subject: [PATCH 07/14] Add transitions, changelog. --- CHANGELOG.md | 1 + specification/metrics/datamodel.md | 10 ++++++++-- 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 67be2664bfd..faceb8d7633 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,7 @@ release. - Rename `gcp_gke` to `gcp_kubernetes_engine` to have consistency with other Google products under `cloud.infrastructure_service` ([#1496](https://github.com/open-telemetry/opentelemetry-specification/pull/1496)) - `http.url` MUST NOT contain credentials ([#1502](https://github.com/open-telemetry/opentelemetry-specification/pull/1502)) +- Adds new metric data model specification ([#1512](https://github.com/open-telemetry/opentelemetry-specification/pull/1512)) ## v1.0.1 (2021-02-11) diff --git a/specification/metrics/datamodel.md b/specification/metrics/datamodel.md index 70de49dd800..200fe522e83 100644 --- a/specification/metrics/datamodel.md +++ b/specification/metrics/datamodel.md @@ -119,7 +119,13 @@ decisions within the metrics data model. ## Model Details -TODO(jsuereth): Recap split of 3 pieces +OpenTelemetry fragments metrics into three interacting models: + +- An Event model, representing how instrumentation reports metric data. +- A TimeSeries model, representing how backends store metric data. +- The *O*pen*T*e*L*emetry *P*rotocol (OTLP) data model representing how metrics + are manipulated and transmitted between the Event model and the TimeSeries + storage. ### Event Model @@ -174,7 +180,7 @@ is no delta temporality in the timeseries model. To precisely define presence and absence of data requires further development of the correspondence between these models. -### OpenTelemetry data model +### OpenTelemetry Protocol data model The OpenTelemetry data model for metrics includes four basic point kinds, all of which satisfy the requirements above, meaning they define a decomposable From 8d0db7a36a6b9f91652a11211d4146b38e36952b Mon Sep 17 00:00:00 2001 From: Josh Suereth Date: Wed, 10 Mar 2021 12:20:12 -0500 Subject: [PATCH 08/14] Add out-of-scope use cases. --- specification/metrics/datamodel.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/specification/metrics/datamodel.md b/specification/metrics/datamodel.md index 200fe522e83..5631eddcf7c 100644 --- a/specification/metrics/datamodel.md +++ b/specification/metrics/datamodel.md @@ -117,6 +117,21 @@ breadth of OTel metrics usage. These are considered the "core" use-cases used to analyze tradeoffs and design decisions within the metrics data model. + +### Out of Scope Use-cases + +The metrics data model is NOT designed to be a perfect rosetta stone of metrics. +Here are a set of use cases that, while won't be outright unsupported, are not +in scope for key design decisions: + +- Using OTLP as an intermediary format between two non-compatible formats + - Importing [statsd](https://github.com/statsd/statsd) => Prometheus PRW + - Importing [collectd](https://collectd.org/wiki/index.php/Binary_protocol#:~:text=The%20binary%20protocol%20is%20the,some%20documentation%20to%20reimplement%20it) + => Prometheus PRW + - Importing Prometheus endpoint scrape => [statsd push | collectd | opencensus] + - Importing OpenCensus "oca" => any non OC or OTel format +- TODO: define others. + ## Model Details OpenTelemetry fragments metrics into three interacting models: From f157ed915ad7bbba104aa035532926d21b68ea93 Mon Sep 17 00:00:00 2001 From: Josh Suereth Date: Wed, 10 Mar 2021 17:21:16 +0000 Subject: [PATCH 09/14] Update specification/metrics/datamodel.md Co-authored-by: Aaron Abbott --- specification/metrics/datamodel.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/metrics/datamodel.md b/specification/metrics/datamodel.md index 5631eddcf7c..7b4216b54ba 100644 --- a/specification/metrics/datamodel.md +++ b/specification/metrics/datamodel.md @@ -177,7 +177,7 @@ consisting of several metadata properties: - Metric name and description - Label set -- Kind of point +- Kind of point (integer, floating point, etc) - Unit of measurement The primary data of each timeseries are ordered (timestamp, value) points, for From 7fd0ad2021f96885f040b0e15d673eb750c49c49 Mon Sep 17 00:00:00 2001 From: Josh Suereth Date: Wed, 10 Mar 2021 12:27:32 -0500 Subject: [PATCH 10/14] Updates from review. --- specification/metrics/datamodel.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specification/metrics/datamodel.md b/specification/metrics/datamodel.md index 7b4216b54ba..3e90273bf9e 100644 --- a/specification/metrics/datamodel.md +++ b/specification/metrics/datamodel.md @@ -165,10 +165,10 @@ three categories: instruments are primarily useful for monitoring a rate value, whereas non-monotonic instruments are primarily useful for monitoring a total value. -In this model, the primary data are (instrument, number) points, originally +In the Event model, the primary data are (instrument, number) points, originally observed in real time or on demand (for the synchronous and asynchronous cases, respectively). The instruments and model use-cases will be described in greater -detail as we link this model with the other two. +detail as we link the event model with the other two. ### Timeseries Model From 8895928c60a2dcd9e995ec1e8db82c33f6e39d4f Mon Sep 17 00:00:00 2001 From: Josh Suereth Date: Wed, 10 Mar 2021 12:29:12 -0500 Subject: [PATCH 11/14] Fix lint issue. --- specification/metrics/datamodel.md | 1 - 1 file changed, 1 deletion(-) diff --git a/specification/metrics/datamodel.md b/specification/metrics/datamodel.md index 3e90273bf9e..2968f03ee69 100644 --- a/specification/metrics/datamodel.md +++ b/specification/metrics/datamodel.md @@ -117,7 +117,6 @@ breadth of OTel metrics usage. These are considered the "core" use-cases used to analyze tradeoffs and design decisions within the metrics data model. - ### Out of Scope Use-cases The metrics data model is NOT designed to be a perfect rosetta stone of metrics. From 517b6544253ba9003b1864e73ca8b8e000a4cc9e Mon Sep 17 00:00:00 2001 From: Josh Suereth Date: Fri, 12 Mar 2021 16:49:14 +0000 Subject: [PATCH 12/14] Update specification/metrics/datamodel.md Co-authored-by: Reiley Yang --- specification/metrics/datamodel.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/metrics/datamodel.md b/specification/metrics/datamodel.md index 2968f03ee69..f507ab4ea27 100644 --- a/specification/metrics/datamodel.md +++ b/specification/metrics/datamodel.md @@ -151,7 +151,7 @@ anchor our understanding of the OpenTelemetry data model and are divided into three categories: - Synchronous vs. Asynchronous. The act of calling a Metrics API in a - synchronous context means the application calls the SDK, typically having + synchronous context means the application/library calls the SDK, typically having associated trace context and baggage; an Asynchronous instrument is called at collection time, through a callback, and lacks context. - Adding vs. Grouping. Whereas adding instruments express a sum, grouping From b844bc18e6369d23c3a3c18954b26cb159e8b828 Mon Sep 17 00:00:00 2001 From: Joshua MacDonald Date: Mon, 15 Mar 2021 14:21:03 -0700 Subject: [PATCH 13/14] Update specification/metrics/datamodel.md Co-authored-by: Reiley Yang --- specification/metrics/datamodel.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/metrics/datamodel.md b/specification/metrics/datamodel.md index f507ab4ea27..f40d0705050 100644 --- a/specification/metrics/datamodel.md +++ b/specification/metrics/datamodel.md @@ -160,7 +160,7 @@ three categories: to grouping instruments are generally not. Adding instrument values are always parts of a sum, while grouping instrument values are individual measurements. - Monotonic vs. Non-Monotonic. The adding instruments are categorized by whether - the derivative of the quantity they express is non-negative. Monotonic + the derivative of the quantity they express is non-negative. Monotonic instruments are primarily useful for monitoring a rate value, whereas non-monotonic instruments are primarily useful for monitoring a total value. From aef7a65411b99a27f526f69344e9df83df770d43 Mon Sep 17 00:00:00 2001 From: Josh Suereth Date: Tue, 16 Mar 2021 14:08:05 -0400 Subject: [PATCH 14/14] Updated from review. --- specification/metrics/datamodel.md | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/specification/metrics/datamodel.md b/specification/metrics/datamodel.md index 2968f03ee69..69f08235ea7 100644 --- a/specification/metrics/datamodel.md +++ b/specification/metrics/datamodel.md @@ -99,20 +99,23 @@ breadth of OTel metrics usage. temporality for a stateless client, stateful server: - Collector re-aggregates into 60 second resolution - Collector converts delta to cumulative temporality -3. A number of OTel SDKs running locally each exports 10 second resolution, each +3. OTel SDK exports both 10 seconds resolution (e.g. CPU, request latency) and + 15 minutes resolution (e.g. room temperature) to a single OTel Collector. + The collector exports streams upstream with or without aggregation. +4. A number of OTel SDKs running locally each exports 10 second resolution, each reports to a single (local) OTel collector. - Collector re-aggregates into 60 second resolution - Collector re-aggregates to eliminate the identity of individual SDKs (e.g., distinct `service.instance.id` values) - Collector outputs to an OTLP destination -4. Pool of OTel collectors receive OTLP and export Prometheus Remote Write +5. Pool of OTel collectors receive OTLP and export Prometheus Remote Write - Collector joins service discovery with metric resources - Collector computes “up”, staleness marker - Collector applies a distinct external label -5. OTel collector receives Statsd and exports OTLP +6. OTel collector receives Statsd and exports OTLP - With delta temporality: stateless collector - With cumulative temporality: stateful collector -6. OTel SDK exports directly to 3P backend +7. OTel SDK exports directly to 3P backend These are considered the "core" use-cases used to analyze tradeoffs and design decisions within the metrics data model. @@ -189,10 +192,10 @@ three value types: This model may be viewed as an idealization of [Prometheus Remote Write](https://docs.google.com/document/d/1LPhVRSFkGNSuU1fBd81ulhsCPR4hkSZyyBj1SZ8fWOM/edit#heading=h.3p42p5s8n0ui). Like that protocol, we are additionally concerned with knowing when a point -value is defined, as compared with being implicitly or explicitly absent. There -is no delta temporality in the timeseries model. To precisely define presence -and absence of data requires further development of the correspondence between -these models. +value is defined, as compared with being implicitly or explicitly absent. A +metric stream of delta data points defines time-interval values, not +point-in-time values. To precisely define presence and absence of data requires +further development of the correspondence between these models. ### OpenTelemetry Protocol data model