feat: merge user supplied and default plugin configs

[naseemkullah]

fixes #616

Which problem is this PR solving?

By merging user supplied config two levels deep, a user can now only configure the diff
from the default auto enablement of installed plugins, rather than
having to explicitly re enable all plugins if at least one of their
configurations require are edited.

Furthermore, user supplied plugins not listed in default plugins are
implicitly enabled.

Short description of the changes

1. User supplied config is checked for any default plugin configs, those are merged with the default config of default plugins.

2. User supplied plugins which are not part of the default plugins list are implicitly enabled, but can be disabled with enabled: false property.

3. The merged user supplied plugins are then merged with the default plugins to include any default plugins that may not have been configured by the user.

[dyladan]

Can you please add tests

[codecov-io]

Codecov Report

Merging #980 into master will increase coverage by 0.02%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #980      +/-   ##
==========================================
+ Coverage   92.34%   92.36%   +0.02%     
==========================================
  Files         115      115              
  Lines        3398     3407       +9     
  Branches      684      686       +2     
==========================================
+ Hits         3138     3147       +9     
  Misses        260      260              

Impacted Files	Coverage Δ
...kages/opentelemetry-node/src/NodeTracerProvider.ts	100.00% <100.00%> (ø)

[markwolff]

Is enabled a required plugin config? If not, I think it would be a good field to add.

[naseemkullah]

Seems like it is not required:

opentelemetry-js/packages/opentelemetry-api/src/trace/instrumentation/Plugin.ts

Lines 56 to 60 in b1fe2c8

	/**
	* Whether to enable the plugin.
	* @default true
	*/
	enabled?: boolean;

But it also seems to default to true but I'm not sure why/how that gets lost. That's why I am making user supplied plugins (that are not part of the default plugin list) enabled parameter default to true here.

[naseemkullah]

bump

[vmarchaud]

I think we should add a specific test that assert that the default options are not overriden by the user one to be sure we dont introduce a regression in the future

[dyladan]

I think we should add a specific test that assert that the default options are not overriden by the user one to be sure we dont introduce a regression in the future

Default options should be overridden by user options.

[vmarchaud]

@dyladan Sorry i meant the fact that the user pass its custom options override all default options

[dyladan]

@naseemkullah sorry this has been such a long process. I agree with @vmarchaud we should probably have a test that if user supplies config for http, the grpc plugin for instance does not have it's config mangled or overridden.

[naseemkullah]

np thanks for the feedback @dyladan @vmarchaud , I'll work on getting the test and let you know if any blockers come up (fetching the plugin config and being able to check it in a test for grpc for example).

[naseemkullah]

Alright I added a test by checking if grpc plugin would successfully load without any grpc related config in a situation with user supplied plugin config.

Furthermore, not sure how to incorporate this in test, but i console logged {plugins}:

{
  plugins: [
    SimpleModulePlugin {
      _tracerName: undefined,
      _tracerVersion: undefined,
      moduleName: 'simple-module',
      _moduleExports: [Object],
      _tracer: [Tracer],
      _logger: NoopLogger {},
      _internalFilesExports: {},
      _config: [Object]
    },
...
...
...
    GrpcPlugin {
      _tracerName: '@opentelemetry/plugin-grpc',
      _tracerVersion: '0.8.0',
      moduleName: 'grpc',
      version: '1.24.2',
      supportedVersions: [Array],
      _internalFilesList: [Object],
      _basedir: '/open-telemetry/opentelemetry-js/packages/opentelemetry-node/node_modules/grpc',
      _config: [Object],
      _moduleExports: [Object],
      _tracer: [Tracer],
      _logger: NoopLogger {},
      _internalFilesExports: [Object]
    }
  ]
}

[naseemkullah]

I am having issues running lint and lint:fix locally, might have something to do with recent migration to eslint, any ideas @dyladan @mayurkale22 ?

Oops! Something went wrong! :(

ESLint: 6.8.0.

ESLint couldn't find the plugin "eslint-plugin-header".

(The package "eslint-plugin-header" was not found when loaded as a Node module from the directory "/Users/naseemullah/repos/github.com/open-telemetry/opentelemetry-js/packages/opentelemetry-node".)

It's likely that the plugin isn't installed correctly. Try reinstalling by running the following:

    npm install eslint-plugin-header@latest --save-dev

In any case i will adjust lint errors manually. This should be ready to go upon build success.

[dyladan]

I would first try to do a full clean and rebuild as the dependencies are all very different

lerna clean -y && rm -rf node_modules && npm install

[naseemkullah]

```shell
lerna clean -y && rm -rf node_modules && npm install

tried from the opentelemetry-node folder with no luck trying from root folder...

[dyladan]

tried from the opentelemetry-node folder with no luck trying from root folder...

This should be run from the root folder. If you're on windows you may need to split it into separate commands:

lerna clean -y
# somehow delete node_modules idk windows :P
npm install

[naseemkullah]

tried from the opentelemetry-node folder with no luck trying from root folder...

This should be run from the root folder. If you're on windows you may need to split it into separate commands:

lerna clean -y
# somehow delete node_modules idk windows :P
npm install

Ah ok, so from root folder should work.

I've been living Windows free since my last enterprise gig a few years back!
Apart from when I help my dad with his computer :D

[naseemkullah]

Seem to be getting errors on non related code:

> @opentelemetry/context-zone-peer-dep@0.8.1 lint /root/project/packages/opentelemetry-context-zone-peer-dep
> eslint . --ext .ts


/root/project/packages/opentelemetry-context-zone-peer-dep/src/ZoneContextManager.ts
   58:35  error    Don't use `Function` as a type. The `Function` type accepts any function-like value.
It provides no type safety when calling the function, which can be a common source of bugs.
It also accepts things like class declarations, which will throw at runtime as they will not be called with `new`.
If you are expecting the function to accept certain arguments, you should explicitly define the function shape  @typescript-eslint/ban-types
   60:44  warning  Unexpected any. Specify a different type                                                                                                                                                                                                                                                                                                                                                                            @typescript-eslint/no-explicit-any
  139:15  error    Don't use `Function` as a type. The `Function` type accepts any function-like value.
It provides no type safety when calling the function, which can be a common source of bugs.
It also accepts things like class declarations, which will throw at runtime as they will not be called with `new`.
If you are expecting the function to accept certain arguments, you should explicitly define the function shape  @typescript-eslint/ban-types
  144:12  warning  Missing return type on function                                                                                                                                                                                                                                                                                                                                                                                     @typescript-eslint/explicit-module-boundary-types
  144:12  warning  Argument 'opts' should be typed                                                                                                                                                                                                                                                                                                                                                                                     @typescript-eslint/explicit-module-boundary-types
  145:13  error    Don't use `{}` as a type. `{}` actually means "any non-nullish value".
- If you want a type meaning "any object", you probably want `Record<string, unknown>` instead.
- If you want a type meaning "any value", you probably want `unknown` instead                                                                                                                                                                @typescript-eslint/ban-types
  148:14  warning  Unexpected any. Specify a different type                                                                                                                                                                                                                                                                                                                                                                            @typescript-eslint/no-explicit-any
  172:15  error    Don't use `Function` as a type. The `Function` type accepts any function-like value.
It provides no type safety when calling the function, which can be a common source of bugs.
It also accepts things like class declarations, which will throw at runtime as they will not be called with `new`.
If you are expecting the function to accept certain arguments, you should explicitly define the function shape  @typescript-eslint/ban-types
  174:12  warning  Missing return type on function                                                                                                                                                                                                                                                                                                                                                                                     @typescript-eslint/explicit-module-boundary-types
  174:28  error    Don't use `{}` as a type. `{}` actually means "any non-nullish value".
- If you want a type meaning "any object", you probably want `Record<string, unknown>` instead.
- If you want a type meaning "any value", you probably want `unknown` instead                                                                                                                                                                @typescript-eslint/ban-types

/root/project/packages/opentelemetry-context-zone-peer-dep/src/types.ts
  26:25  warning  Unexpected any. Specify a different type  @typescript-eslint/no-explicit-any
  28:6   warning  Unexpected any. Specify a different type  @typescript-eslint/no-explicit-any
  31:25  warning  Unexpected any. Specify a different type  @typescript-eslint/no-explicit-any
  33:6   warning  Unexpected any. Specify a different type  @typescript-eslint/no-explicit-any

✖ 14 problems (5 errors, 9 warnings)

[dyladan]

Those are warnings not errors that you posted. The warnings don't fail the build

[mayurkale22]

Seem to be getting errors on non related code:

/root/project/packages/opentelemetry-context-zone-peer-dep/src/types.ts
  26:25  warning  Unexpected any. Specify a different type  @typescript-eslint/no-explicit-any
  28:6   warning  Unexpected any. Specify a different type  @typescript-eslint/no-explicit-any
  31:25  warning  Unexpected any. Specify a different type  @typescript-eslint/no-explicit-any
  33:6   warning  Unexpected any. Specify a different type  @typescript-eslint/no-explicit-any

✖ 14 problems (5 errors, 9 warnings)

#1093 is created to address these warnings. Please ignore it for now.

[naseemkullah]

Those are warnings not errors that you posted. The warnings don't fail the build

Ah right but still seems to be the case, message updated to include more context

[naseemkullah]

There are errors as well, npm run lint finally works again locally, changes in question are linted and I think this is good to go if we ignore the errors from zoneContextMgr

[dyladan]

hmm. the question is why does it not break the build for others...

[mayurkale22]

$ git clean -dfx
$ npm install
$ npm run lint

I tried these commands locally and everything is working well for me.

[naseemkullah]

now

lerna ERR! npm run lint exited 1 in '@opentelemetry/context-async-hooks'
lerna ERR! npm run lint stdout:

> @opentelemetry/context-async-hooks@0.8.2 lint /root/project/packages/opentelemetry-context-async-hooks
> eslint . --ext .ts


/root/project/packages/opentelemetry-context-async-hooks/src/AsyncHooksContextManager.ts
   46:20  error    Expected a `const` instead of a literal type assertion                                                                                                                                                                                                                                                                                                                                                              @typescript-eslint/prefer-as-const
   47:11  error    Expected a `const` instead of a literal type assertion                                                                                                                                                                                                                                                                                                                                                              @typescript-eslint/prefer-as-const
   48:13  error    Expected a `const` instead of a literal type assertion                                                                                                                                                                                                                                                                                                                                                              @typescript-eslint/prefer-as-const
   49:24  error    Expected a `const` instead of a literal type assertion                                                                                                                                                                                                                                                                                                                                                              @typescript-eslint/prefer-as-const
   50:28  error    Expected a `const` instead of a literal type assertion                                                                                                                                                                                                                                                                                                                                                              @typescript-eslint/prefer-as-const
   95:37  warning  Unexpected any. Specify a different type                                                                                                                                                                                                                                                                                                                                                                            @typescript-eslint/no-explicit-any
  144:35  error    Don't use `Function` as a type. The `Function` type accepts any function-like value.
It provides no type safety when calling the function, which can be a common source of bugs.
It also accepts things like class declarations, which will throw at runtime as they will not be called with `new`.
If you are expecting the function to accept certain arguments, you should explicitly define the function shape  @typescript-eslint/ban-types
  146:44  error    Don't use `{}` as a type. `{}` actually means "any non-nullish value".
- If you want a type meaning "any object", you probably want `Record<string, unknown>` instead.
- If you want a type meaning "any value", you probably want `unknown` instead                                                                                                                                                                @typescript-eslint/ban-types
  159:30  warning  Unexpected any. Specify a different type                                                                                                                                                                                                                                                                                                                                                                            @typescript-eslint/no-explicit-any
  205:67  error    Don't use `Function` as a type. The `Function` type accepts any function-like value.
It provides no type safety when calling the function, which can be a common source of bugs.
It also accepts things like class declarations, which will throw at runtime as they will not be called with `new`.
If you are expecting the function to accept certain arguments, you should explicitly define the function shape  @typescript-eslint/ban-types
  206:12  warning  Missing return type on function                                                                                                                                                                                                                                                                                                                                                                                     @typescript-eslint/explicit-module-boundary-types
  206:28  error    Don't use `{}` as a type. `{}` actually means "any non-nullish value".
- If you want a type meaning "any object", you probably want `Record<string, unknown>` instead.
- If you want a type meaning "any value", you probably want `unknown` instead                                                                                                                                                                @typescript-eslint/ban-types
  227:15  error    Don't use `Function` as a type. The `Function` type accepts any function-like value.
It provides no type safety when calling the function, which can be a common source of bugs.
It also accepts things like class declarations, which will throw at runtime as they will not be called with `new`.
If you are expecting the function to accept certain arguments, you should explicitly define the function shape  @typescript-eslint/ban-types
  229:12  warning  Missing return type on function                                                                                                                                                                                                                                                                                                                                                                                     @typescript-eslint/explicit-module-boundary-types
  229:28  error    Don't use `{}` as a type. `{}` actually means "any non-nullish value".
- If you want a type meaning "any object", you probably want `Record<string, unknown>` instead.
- If you want a type meaning "any value", you probably want `unknown` instead                                                                                                                                                                @typescript-eslint/ban-types
  250:15  error    Don't use `Function` as a type. The `Function` type accepts any function-like value.
It provides no type safety when calling the function, which can be a common source of bugs.
It also accepts things like class declarations, which will throw at runtime as they will not be called with `new`.
If you are expecting the function to accept certain arguments, you should explicitly define the function shape  @typescript-eslint/ban-types
  254:12  warning  Missing return type on function                                                                                                                                                                                                                                                                                                                                                                                     @typescript-eslint/explicit-module-boundary-types
  254:28  error    Don't use `{}` as a type. `{}` actually means "any non-nullish value".
- If you want a type meaning "any object", you probably want `Record<string, unknown>` instead.
- If you want a type meaning "any value", you probably want `unknown` instead                                                                                                                                                                @typescript-eslint/ban-types

✖ 18 problems (13 errors, 5 warnings)
  5 errors and 0 warnings potentially fixable with the `--fix` option.

[dyladan]

Should be fixed with #1094

[dyladan]

I think it is most likely a versioning discrepancy as the lint versions are not pinned in CI. #1081 also fixes this shortcoming.

[rlibm]

Trying to confirm that the regression in the plugin load behaviour is intentional or not. My intention is to have a few plugins loaded for instrumentation. I have been loading an array of plugins to the NodeTraceProvider with code like (yes, I have my own set of default plugins which covers http, https, express, and some proprietary modules):

    const plugins = composePlugins(options.plugins || defaultPlugins, options.pluginOptions || {});
    const config = {
      plugins,
      sampler: ALWAYS_SAMPLER,
      traceParams: {
        numberOfEventsPerSpan: 64,
        numberOfAttributesPerSpan: 64
      },
      logger
    };

    this.#provider = new NodeTracerProvider(config);

We are intentionally not instrumenting some other libraries, for example, ioredis.

With this change, we are getting errors on startup:

ERROR PluginLoader#load: could not load plugin @opentelemetry/plugin-ioredis of module ioredis. Error: Cannot find module '@opentelemetry/plugin-ioredis'
Require stack:
- /home/rolf/bmix/platapi-core/node_modules/@opentelemetry/node/build/src/instrumentation/PluginLoader.js
- /home/rolf/bmix/platapi-core/node_modules/@opentelemetry/node/build/src/NodeTracerProvider.js
- /home/rolf/bmix/platapi-core/node_modules/@opentelemetry/node/build/src/index.js
......

I see now that all the DEFAULT_INSTRUMENTATION_PLUGINS plugins are enabled.

I would like to remove all of these default plugins, or disable them all, but I cannot see an exposed way to do that.

3 questions:

1. was it intentional to change the behaviour this way, and will it stay this way?
2. how do I get the list of default plugins so I can run through them and disable them all?
3. is there a better way?

For the moment, I am doing this:

// See https://github.com/open-telemetry/opentelemetry-js/pull/980
const DEFAULT_DISABLE_HACK = {
  mongodb: { enabled: false },
  grpc: { enabled: false },
  http: { enabled: false },
  https: { enabled: false },
  mysql: { enabled: false },
  pg: { enabled: false },
  redis: { enabled: false },
  ioredis: { enabled: false },
  'pg-pool': { enabled: false },
  express: { enabled: false }
};

.....

    const useplugins = composePlugins(options.plugins || defaultPlugins, options.pluginOptions || {});

    const plugins = { ...DEFAULT_DISABLE_HACK, ...useplugins };

    const config = {
      plugins,
      sampler: ALWAYS_SAMPLER,
      traceParams: {
        numberOfEventsPerSpan: 256,
        numberOfAttributesPerSpan: 256
      },
      logger
    };

    this.#provider = new NodeTracerProvider(config);

[naseemkullah]

Trying to confirm that the regression in the plugin load behaviour is intentional or not. My intention is to have a few plugins loaded for instrumentation. I have been loading an array of plugins to the NodeTraceProvider with code like (yes, I have my own set of default plugins which covers http, https, express, and some proprietary modules):

    const plugins = composePlugins(options.plugins || defaultPlugins, options.pluginOptions || {});
    const config = {
      plugins,
      sampler: ALWAYS_SAMPLER,
      traceParams: {
        numberOfEventsPerSpan: 64,
        numberOfAttributesPerSpan: 64
      },
      logger
    };

    this.#provider = new NodeTracerProvider(config);

We are intentionally not instrumenting some other libraries, for example, ioredis.

With this change, we are getting errors on startup:

ERROR PluginLoader#load: could not load plugin @opentelemetry/plugin-ioredis of module ioredis. Error: Cannot find module '@opentelemetry/plugin-ioredis'
Require stack:
- /home/rolf/bmix/platapi-core/node_modules/@opentelemetry/node/build/src/instrumentation/PluginLoader.js
- /home/rolf/bmix/platapi-core/node_modules/@opentelemetry/node/build/src/NodeTracerProvider.js
- /home/rolf/bmix/platapi-core/node_modules/@opentelemetry/node/build/src/index.js
......

I see now that all the DEFAULT_INSTRUMENTATION_PLUGINS plugins are enabled.

I would like to remove all of these default plugins, or disable them all, but I cannot see an exposed way to do that.

3 questions:

1. was it intentional to change the behaviour this way, and will it stay this way?
2. how do I get the list of default plugins so I can run through them and disable them all?
3. is there a better way?

Hi @rlibm

1. It was, and as such I think it will stay that way, provided your ux can be improved.
2. I know of 3 ways:

• README
• source code
• Logs when starting your process to see what plugins that were not explicitly enabled are trying to load.

3. With that said we could surely reduce the severity of

ERROR PluginLoader#load: could not load plugin @opentelemetry/plugin-ioredis of module ioredis. Error: Cannot find module '@opentelemetry/plugin-ioredis'

to something like

WARN PluginLoader#load: could not load plugin @opentelemetry/plugin-ioredis of module ioredis. Error: Cannot find module '@opentelemetry/plugin-ioredis', if instrumentation of this module is undesired, please ignore.

Or add yet another config option equivalent to your DEFAULT_DISABLE_HACK, maybe a boolean like mergeDefault

@dyladan @mayurkale22 thoughts comments or corrections to the above?

[rlibm]

The specific issue I have is this code is being put in a library that wraps opentelemetry, and fortunately for me, I noticed the error before it went further. I am fortunate that the first system I tested it on had ioredis installed, and threw that error, and I am also fortunate that the plugin-ioredis was not included in the dependencies.

Otherwise the ioredis and other plugins may have been silently, and unexpectedly loaded, and caused significant increases in tracing output, and possibly issues in down-stream automated anomoly detection, etc.

In effect, the risk I have is 2-fold:

1. I have to continuously monitor whether the default plugin list has changed in each code-release of opentelemetry, disabling plugins as they arrive....
2. I need positive assurance that only the plugins I want to load, are loaded.

I would recommend some option on the NodeTracingProvider that bypasses the merge, and uses the supplied plugins as-configured.

Automatic merging with unknown content is a bad solution in enterprise environments.

This includes not just completely new plugins, but also options on existing plugins. Changing the options, or adding new options on default plugins could cause significant regressions in existing applicatoins when new (even minor) versions of OpenTelemetry are released.

[naseemkullah]

The specific issue I have is this code is being put in a library that wraps opentelemetry, and fortunately for me, I noticed the error before it went further. I am fortunate that the first system I tested it on had ioredis installed, and threw that error, and I am also fortunate that the plugin-ioredis was not included in the dependencies.

Otherwise the ioredis and other plugins may have been silently, and unexpectedly loaded, and caused significant increases in tracing output, and possibly issues in down-stream automated anomoly detection, etc.

In effect, the risk I have is 2-fold:

1. I have to continuously monitor whether the default plugin list has changed in each code-release of opentelemetry, disabling plugins as they arrive....
2. I need positive assurance that only the plugins I want to load, are loaded.

I would recommend some option on the NodeTracingProvider that bypasses the merge, and uses the supplied plugins as-configured.

Automatic merging with unknown content is a bad solution in enterprise environments.

Is it fortune to not have plugin-ioredis in the dependencies or a desire to not instrument ioredis? 😛

In any case, I think a mergeDefault option (enabled by default 🙏 👼 ) would suit you/enterprise environments. How's about a PR? 😁

[rlibm]

We have the desire to not instrument ioredis. for cloud.ibm.com the redis volumes are huge. Inadvertently enabling it would be .... problematic.

Side-Note, we're migrating from OpenCensus to OpenTelemetry. The OpenTelemetry is really close to production (yes, we know this library is not yet out of Beta).

I will work on a PR, but we natively use JavaScript, I don't have tooling set up for .ts as such. May take a while.

[naseemkullah]

@rlibm Could you elaborate on the downsides of instrumenting a huge redis dataset? Do you think it would add latency and if so would it be possible to do some benchmarking? If it's a question of data produced, there are sampling strategies to use, also FYI this PR prevents random background jobs from being traced, and only redis ops that have a parent span will be traced.

That is exciting that IBM Cloud is migrating to OTEL! 🚀

FWIW We are starting to use this quite a bit in production at Transit, but we are a startup 😄

Great to hear you are up for a PR! 👍 TS is great, you're gonna love it... and no rush, we'll be waiting! 😄

[naseemkullah]

ooh and one more thing, don't be a stranger if you encounter any time wasters/blockers when working on the PR https://gitter.im/open-telemetry/opentelemetry-node