feat(exporters)!: rewrite exporter config logic #4971

pichlermarc · 2024-09-02T14:39:28Z

Which problem is this PR solving?

Config code for exporters (user-provided, defaults, and env config) is currently hard to test individually and is spread/duplicated all over the different signal exporters.

This PR moves the configuration as much as possible into separate files/functions and splits them up into:

defaults
fallback merging (user-provided merging with env var config and defaults)
- fallback merging is intentionally held modular in an effort to possibly allow a file-config fallback in the future
parsing the config from the environment

Since most config is now moved to base-packages we can remove the in-depth tests from the specific packages. This reduces duplication and makes further changes easier, as they will touch "fewer" files. Since now the configuration-code is not part of the individual exporter-implementations, collapsing the base exporters into one class becomes simpler as we have to worry less about the inheritance structure (abstract methods were removed in this PR as they were config-related).

In addition to these improvements, this PR also fixes #3748, #3747 which was a side-effect of the previous way of implementing env var config.

Breaking changes:

All exporters:
- (user-facing) getDefaultUrl was intended for internal use has been removed from all exporters
- (user-facing) getUrlFromConfig was intended for internal use and has been removed from all exporters
- (user-facing) hostname was intended for internal use and has been removed from all exporters
- (user-facing) url was intended for internal use and has been removed from all exporters
- (user-facing) timeoutMillis was intended for internal use and has been removed from all exporters
- (user-facing) onInit was intended for internal use and has been removed from all exporters
Exporter base package:
- (internal) parseHeaders is now not exported anymore
- (internal) appendResourcePathToUrl is now not exported anymore
- (internal) appendResourcePathToUrlIfNeeded is now not exported anymore
- (internal) configureExporterTimeout is now not exported anymore
- (internal) invalidTimeout is now not exported anymore

Follow-up steps:

this PR did not apply the same to the metrics-specific enviornment variables like OTEL_EXPORTER_OTLP_TEMPORALITY_PREFERENCE, this will be done in a follow-up.
consolidate all base-classes OTLPExporterNodeBase, OTLPExporterBrowserBase, OTLPGrpcExporterNodeBase into one shared base, that favors composition over inheritance. This can be done by:
- introducing a function that converts the "legacy" (i.e. current) configuration to the new configuration model (introduced in this PR)
- creating a factory function that creates an OTLPExporterDelegate (draft) based on the "new" config, this replaces the OTLPExporterNodeBase, OTLPExporterBrowserBase, OTLPGrpcExporterNodeBase classes which right now do essentially the same, except for the constructor which will be replaced by the factory function.
- final exporters don't inherit from any of these anymore, but they simply implement all functions by calling the the delegate's functions.
clean up existing tests to avoid testing export-output in-depth (this is tested in @opentelemetry/otlp-transformer, testing that the output is the correct format should suffice)

Fixes #3748
Fixes #3747

Type of change

Breaking change (fix or feature that would cause existing functionality to not work as expected)
Bug fix ([otlp-grpc-exporter-base] base package is using trace-specific env vars #3748 and [exporter-logs-otlp-grpc] add support for logs specific env vars #3747)

How Has This Been Tested?

Remaining unit tests
New unit tests

codecov · 2024-09-16T13:11:03Z

Codecov Report

Attention: Patch coverage is 95.59322% with 13 lines in your changes missing coverage. Please review.

Project coverage is 93.26%. Comparing base (f8ab559) to head (e24837d).
Report is 24 commits behind head on main.

Files with missing lines	Patch %	Lines
...se/src/platform/browser/OTLPExporterBrowserBase.ts	20.00%	4 Missing ⚠️
.../src/platform/node/convert-legacy-agent-options.ts	50.00%	4 Missing ⚠️
...grpc-exporter-base/src/OTLPGRPCExporterNodeBase.ts	72.72%	3 Missing ⚠️
...e/src/configuration/otlp-http-env-configuration.ts	95.74%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4971      +/-   ##
==========================================
- Coverage   93.39%   93.26%   -0.14%     
==========================================
  Files          46      317     +271     
  Lines         712     8194    +7482     
  Branches      120     1640    +1520     
==========================================
+ Hits          665     7642    +6977     
- Misses         47      552     +505

Files with missing lines	Coverage Δ
...ges/exporter-logs-otlp-grpc/src/OTLPLogExporter.ts	`100.00% <ø> (ø)`
...ogs-otlp-http/src/platform/node/OTLPLogExporter.ts	`100.00% <100.00%> (ø)`
...gs-otlp-proto/src/platform/node/OTLPLogExporter.ts	`100.00% <100.00%> (ø)`
.../exporter-trace-otlp-grpc/src/OTLPTraceExporter.ts	`100.00% <ø> (ø)`
...e-otlp-http/src/platform/node/OTLPTraceExporter.ts	`100.00% <100.00%> (ø)`
...-otlp-proto/src/platform/node/OTLPTraceExporter.ts	`100.00% <100.00%> (ø)`
...porter-metrics-otlp-grpc/src/OTLPMetricExporter.ts	`100.00% <ø> (ø)`
...-otlp-http/src/platform/node/OTLPMetricExporter.ts	`100.00% <100.00%> (ø)`
...orter-metrics-otlp-proto/src/OTLPMetricExporter.ts	`100.00% <100.00%> (ø)`
...ackages/otlp-exporter-base/src/OTLPExporterBase.ts	`94.44% <ø> (ø)`
... and 13 more

... and 250 files with indirect coverage changes

david-luna

this was a big one :)

I like the approach to have config separated by functions and the merge the results. Just a couple of questions before approving it. Thanks for taking the time to simplify the config

david-luna · 2024-09-26T11:10:31Z

experimental/packages/otlp-exporter-base/src/configuration/otlp-http-env-configuration.ts

+import { getSharedConfigurationFromEnvironment } from './shared-env-configuration';
+import { OtlpHttpConfiguration } from './otlp-http-configuration';
+
+function getHeadersFromEnv(signalIdentifier: string) {


nit: do you think signals are going to grow much in the future that we need to have the identifier as a string type?

Hmm, I don't think they will grow much. I mostly just chose string as it was the easiest choice to work with and it does not require us to export a new type that could be difficult to extend later.

Do you think we should have a SignalIdentifier type? 🤔

one might think is a commodity but with the type the editor helps you to avoid passing a string which does not refer to any Signal. Is not wrong to not have a type here. I'll leave it to you

I think I will leave it as-is - maybe I'm traumatized by making unintentional breaking changes in TypeScript libraries, but I feel like whatever type I can come up with won't be extendable in a safe way later on. 😕

The fact that libraries have to be type-compatible with other versions of themselves is kind of a mind-bender that did not exist in other languages I came from originally. This API will mostly be internal, so I'll opt to leave it like this for now. We can re-visit this choice in a later iteration - I'm planning to have a final round of API review when the whole exporter restructuring is over to work out smaller things like this. 🙂

david-luna · 2024-09-26T11:22:31Z

experimental/packages/otlp-exporter-base/src/configuration/otlp-http-env-configuration.ts

+    if (!url.endsWith('/')) {
+      url = url + '/';
+    }
+    return url + path;


The function is not checking that

url does not already contain a path

the result string is a valid URL, path could contain invalid chars

So the following call is possible

appendResourcePathToUrl('http://foo.com/bar/test.txt', 'v1/logs') // returns 'http://foo.com/bar/test.txt/v1/logs'

I guess we should let the users define the URL they want but I think we should diag.warn or diag.debug the result for debug purposes (please ignore this comment if the diag. is set in a file below, I haven't reached them yet)

Good point about checking that the final URL is valid. 👍 I added additional checks and logs to that function. I settled on warn as it's very likely that it'll not be able to continue with an invalid URL. 5d0aae1

It's actually intended to append if it does not contain a path already - this is specified behavior for the OTEL_EXPORTER_OTLP_ENDPOINT env var where this function is used. See https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/exporter.md#endpoint-urls-for-otlphttp

For signal-specific ones like OTEL_EXPORTER_OTLP_ENDPOINT no path is actually appended - just the root path if none is there already that's the function above this one 🙂

david-luna · 2024-09-26T13:11:34Z

...mental/packages/otlp-exporter-base/test/common/configuration/otlp-http-configuration.test.ts

+      );
+      assert.deepStrictEqual(config.headers, {
+        foo: 'foo-user-provided',
+        baz: 'null',


bar: undefined does not appear in the result but baz: null does and it's stringified. Any reason for that?

This is mostly to keep the same behavior of the current implementation, which also handles it this way 🙂

undefined

bar: undefined is dropped as calls with http module will throw when there's an undefined header value:

> const http = require('http'); undefined > http.get({host: 'example.com', headers: {'foo': undefined}}) Uncaught: TypeError [ERR_HTTP_INVALID_HEADER_VALUE]: Invalid value "undefined" for header "foo" at __node_internal_captureLargerStackTrace (node:internal/errors:496:5) at new NodeError (node:internal/errors:405:5) at __node_internal_ (node:_http_outgoing:624:11) at ClientRequest.setHeader (node:_http_outgoing:651:3) at new ClientRequest (node:_http_client:291:14) at request (node:http:100:10) at Object.get (node:http:111:15) { code: 'ERR_HTTP_INVALID_HEADER_VALUE' }

We've had users in the past that have ended up with undefined header values as they set a header to process.env.MY_HEADER_ENV_VAR, did not set that env var and then ended up with an exporter that would constantly fail.

null

null is stringified as to ensure it matches the type for Record<string, string> for the headers. Plus it gets stringified to null anyway when it's sent via with the http module so we make it happen earlier to make the type a bit more simple to work with down the chain as we don't have to worry about null values. It's also the current behavior of the exporters, so I tried not to change it.

You can try sending it to https://webhook.site to see that it ends up the same:

> const https = require('https'); undefined > https.get({host: 'webhook.site', headers: {'foo': null}, path: '/<your-webhook-site-path>'}) > https.get({host: 'webhook.site', headers: {'foo': 'null'}, path: '/<your-webhook-site-path>'})

pichlermarc force-pushed the feat/node-exporter-config-rewrite branch 2 times, most recently from 2b185dc to af05548 Compare September 10, 2024 15:30

pichlermarc force-pushed the feat/node-exporter-config-rewrite branch 2 times, most recently from 9bfaad5 to b93d9db Compare September 16, 2024 14:44

feat(exporters)!: rewrite exporter config logic

3c1756f

pichlermarc force-pushed the feat/node-exporter-config-rewrite branch from 11b5f20 to 3c1756f Compare September 16, 2024 15:31

pichlermarc changed the title ~~feat(exporters)!: rewrite exporter config logic for testability~~ feat(exporters)!: rewrite exporter config logic Sep 16, 2024

pichlermarc marked this pull request as ready for review September 16, 2024 15:40

pichlermarc requested review from a team September 16, 2024 15:40

pichlermarc added 3 commits September 20, 2024 13:00

Merge branch 'main' into feat/node-exporter-config-rewrite

0423bde

Merge branch 'main' into feat/node-exporter-config-rewrite

7c3c108

Merge branch 'main' into feat/node-exporter-config-rewrite

b12cf4a

david-luna reviewed Sep 26, 2024

View reviewed changes

david-luna approved these changes Sep 27, 2024

View reviewed changes

pichlermarc added 3 commits September 27, 2024 17:11

fix(otlp-exporter-base): warn when final url is not parsable

5d0aae1

Merge branch 'main' into feat/node-exporter-config-rewrite

6606c0e

Merge branch 'main' into feat/node-exporter-config-rewrite

e24837d

pichlermarc enabled auto-merge September 30, 2024 09:29

pichlermarc added this pull request to the merge queue Sep 30, 2024

Merged via the queue into open-telemetry:main with commit 3007d3e Sep 30, 2024
21 checks passed

pichlermarc deleted the feat/node-exporter-config-rewrite branch September 30, 2024 09:39

opentelemetrybot mentioned this pull request Oct 14, 2024

chore: prepare next release #5068

Merged

pichlermarc mentioned this pull request Oct 15, 2024

TraceExporter for fetch-only environment #4631

Open

This was referenced Nov 7, 2024

maint(deps): bump the otel group in /packages/honeycomb-opentelemetry-web with 8 updates honeycombio/honeycomb-opentelemetry-web#349

Merged

Undocumented Breaking Change in Experimental v0.54.0 #5125

Closed

This was referenced Nov 13, 2024

Configure OTLP metrics exporter with environment variables #3193

Open

feat(otlp-transformer)!: accept ResourceMetrics instead of ResoruceMetrics[] in metrics serializers #5159

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(exporters)!: rewrite exporter config logic #4971

feat(exporters)!: rewrite exporter config logic #4971

pichlermarc commented Sep 2, 2024 •

edited

Loading

codecov bot commented Sep 16, 2024 •

edited

Loading

david-luna left a comment

david-luna Sep 26, 2024

pichlermarc Sep 26, 2024

david-luna Sep 27, 2024

pichlermarc Sep 27, 2024

david-luna Sep 26, 2024

pichlermarc Sep 27, 2024

david-luna Sep 26, 2024

pichlermarc Sep 26, 2024

feat(exporters)!: rewrite exporter config logic #4971

feat(exporters)!: rewrite exporter config logic #4971

Conversation

pichlermarc commented Sep 2, 2024 • edited Loading

Which problem is this PR solving?

Type of change

How Has This Been Tested?

codecov bot commented Sep 16, 2024 • edited Loading

Codecov Report

david-luna left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

undefined

null

pichlermarc commented Sep 2, 2024 •

edited

Loading

codecov bot commented Sep 16, 2024 •

edited

Loading

`undefined`

`null`