[processor/transform] Add Conversion Function to OTTL for Exponential Histo --> Histogram

[daidokoro]

Description

This PR adds a custom metric function to the transformprocessor to convert exponential histograms to explicit histograms.

Link to tracking issue: Resolves #33827

Function Name

convert_exponential_histogram_to_explicit_histogram

Arguments:

• distribution (upper, midpoint, uniform, random)
• ExplicitBoundaries: []float64

Usage example:

processors:
  transform:
    error_mode: propagate
    metric_statements:
    - context: metric
      statements:
        - convert_exponential_histogram_to_explicit_histogram("random", [10.0, 20.0, 30.0, 40.0, 50.0, 60.0, 70.0, 80.0, 90.0, 100.0]) 

Converts:

Resource SchemaURL: 
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope  
Metric #0
Descriptor:
     -> Name: response_time
     -> Description: 
     -> Unit: 
     -> DataType: ExponentialHistogram
     -> AggregationTemporality: Delta
ExponentialHistogramDataPoints #0
Data point attributes:
     -> metric_type: Str(timing)
StartTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2024-07-31 09:35:25.212037 +0000 UTC
Count: 44
Sum: 999.000000
Min: 40.000000
Max: 245.000000
Bucket (32.000000, 64.000000], Count: 10
Bucket (64.000000, 128.000000], Count: 22
Bucket (128.000000, 256.000000], Count: 12
        {"kind": "exporter", "data_type": "metrics", "name": "debug"}

To:

Resource SchemaURL: 
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope  
Metric #0
Descriptor:
     -> Name: response_time
     -> Description: 
     -> Unit: 
     -> DataType: Histogram
     -> AggregationTemporality: Delta
HistogramDataPoints #0
Data point attributes:
     -> metric_type: Str(timing)
StartTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2024-07-30 21:37:07.830902 +0000 UTC
Count: 44
Sum: 999.000000
Min: 40.000000
Max: 245.000000
ExplicitBounds #0: 10.000000
ExplicitBounds #1: 20.000000
ExplicitBounds #2: 30.000000
ExplicitBounds #3: 40.000000
ExplicitBounds #4: 50.000000
ExplicitBounds #5: 60.000000
ExplicitBounds #6: 70.000000
ExplicitBounds #7: 80.000000
ExplicitBounds #8: 90.000000
ExplicitBounds #9: 100.000000
Buckets #0, Count: 0
Buckets #1, Count: 0
Buckets #2, Count: 0
Buckets #3, Count: 2
Buckets #4, Count: 5
Buckets #5, Count: 0
Buckets #6, Count: 3
Buckets #7, Count: 7
Buckets #8, Count: 2
Buckets #9, Count: 4
Buckets #10, Count: 21
        {"kind": "exporter", "data_type": "metrics", "name": "debug"}

Testing

• Several unit tests have been created. We have also tested by ingesting and converting exponential histograms from the statsdreceiver as well as directly via the otlpreceiver over grpc over several hours with a large amount of data.

• We have clients that have been running this solution in production for a number of weeks.

Readme description:

convert_exponential_hist_to_explicit_hist

convert_exponential_hist_to_explicit_hist([ExplicitBounds])

the convert_exponential_hist_to_explicit_hist function converts an ExponentialHistogram to an Explicit (normal) Histogram.

ExplicitBounds is represents the list of bucket boundaries for the new histogram. This argument is required and cannot be empty.

WARNING:

The process of converting an ExponentialHistogram to an Explicit Histogram is not perfect and may result in a loss of precision. It is important to define an appropriate set of bucket boundaries to minimize this loss. For example, selecting Boundaries that are too high or too low may result histogram buckets that are too wide or too narrow, respectively.

[TylerHelmuth]

@daidokoro looks like you need a go mod tidy and then we'll be good.

[daidokoro]

@daidokoro looks like you need a go mod tidy and then we'll be good.

Thanks @TylerHelmuth , executed.

[daidokoro]

@TylerHelmuth

Currently all failing tests don't seem to be related to this PR directly.

• Link check fails, however no links are added by this PR
• Lint fails due to lack of export data in a package that is unrelated to this change
• Another lint job fails with a stream error unrelated to this package
• Lastly the changelog check fails saying there have been changes to the changelog, however this PR does not make changes to that file.

Can you advise on these issues?

Thanks in advance

[TylerHelmuth]

@daidokoro this has been a bad week for our CI as there has been a lot of churning in Core that is causing us to do more core library updates than normal. In addition some new linters are causing issues and the prometheus compliance tests as well. Also the general Go proxy issue.

Things are getting cleaned up. One thing that would help is if you allow us to merge commits into your branch from this PR. That would allow me to update your branch with the latest from main and merge code suggestions I make.

[TylerHelmuth]

Another thing to do is to run make checks locally. Make sure your local Go version is 1.22.7.

[daidokoro]

@daidokoro this has been a bad week for our CI as there has been a lot of churning in Core that is causing us to do more core library updates than normal. In addition some new linters are causing issues and the prometheus compliance tests as well. Also the general Go proxy issue.

Things are getting cleaned up. One thing that would help is if you allow us to merge commits into your branch from this PR. That would allow me to update your branch with the latest from main and merge code suggestions I make.

Thanks @TylerHelmuth for the update. Feel free to make commits and I'll merge all suggestions. Anything else I can do to help let me know.

[TylerHelmuth]

@daidokoro I don't have permission to merge in. I'd say get the latest from main (it should have all the CI fixed), and then ensure make checks

[TylerHelmuth]

Needs a make crosslink