wrong import for "import-in-the-middle"

[matthias-pichler]

What happened?

Steps to Reproduce

Import from "@opentelemetry/instrumentation" and bundle using esbuild with format "cjs"

Expected Result

The build should proceed without warning and not crash during runtime

Actual Result

The build produces an error and crashes during runtime

Additional Details

During build time the following output is shown

▲ [WARNING] Constructing "ImportInTheMiddle" will crash at run-time because it's an import namespace object, not a constructor [call-import-namespace]

    ../../.yarn/__virtual__/@opentelemetry-instrumentation-virtual-0402264686/0/cache/@opentelemetry-instrumentation-npm-0.40.0-a48dab8126-03755fc091.zip/node_modules/@opentelemetry/instrumentation/build/esm/platform/node/instrumentation.js:258:30:
      258 │             var esmHook = new ImportInTheMiddle([module_2.name], { internals: false }, hookFn);
          ╵                               ~~~~~~~~~~~~~~~~~

  Consider changing "ImportInTheMiddle" to a default import instead:

    ../../.yarn/__virtual__/@opentelemetry-instrumentation-virtual-0402264686/0/cache/@opentelemetry-instrumentation-npm-0.40.0-a48dab8126-03755fc091.zip/node_modules/@opentelemetry/instrumentation/build/esm/platform/node/instrumentation.js:48:7:
      48 │ import * as ImportInTheMiddle from 'import-in-the-middle';
         │        ~~~~~~~~~~~~~~~~~~~~~~

OpenTelemetry Setup Code

import {
  diag,
  DiagLogLevel,
  trace,
  Tracer,
  Span,
  SpanKind,
  SpanStatusCode,
} from "@opentelemetry/api";
import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";
import { registerInstrumentations } from "@opentelemetry/instrumentation";
import { AwsLambdaInstrumentation } from "@opentelemetry/instrumentation-aws-lambda";
import { DnsInstrumentation } from "@opentelemetry/instrumentation-dns";
import { AwsInstrumentation } from "@opentelemetry/instrumentation-aws-sdk";
import { HttpInstrumentation } from "@opentelemetry/instrumentation-http";
import { AWSXRayIdGenerator } from "@opentelemetry/id-generator-aws-xray";
import { AWSXRayPropagator } from "@opentelemetry/propagator-aws-xray";
import {
  SemanticAttributes,
  SemanticResourceAttributes,
  CloudProviderValues,
  CloudPlatformValues,
} from "@opentelemetry/semantic-conventions";

  try {
    diag.setLogger(logger as any, DiagLogLevel.ALL);
    const provider = new NodeTracerProvider({
      idGenerator: new AWSXRayIdGenerator(),
    });
    const spanProcessor = new BatchSpanProcessor(
      new OTLPTraceExporter({
        url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT,
      })
    );

    provider.addSpanProcessor(spanProcessor);
    provider.register({
      propagator: new AWSXRayPropagator(),
    });
    trace.setGlobalTracerProvider(provider);

    registerInstrumentations({
      instrumentations: [
        new AwsInstrumentation({}),
        new AwsLambdaInstrumentation({
          requestHook: (span, { event, context }) => {
            const [, , , , accountId] = context.invokedFunctionArn.split(":");
            const functionArn = context.invokedFunctionArn
              .split(":", 7)
              .join(":");

            span.setAttributes({
              [SemanticAttributes.AWS_LAMBDA_INVOKED_ARN]:
                context.invokedFunctionArn,
              [SemanticResourceAttributes.AWS_LOG_GROUP_NAMES]: [
                context.logGroupName,
              ],
              [SemanticResourceAttributes.AWS_LOG_STREAM_NAMES]: [
                context.logStreamName,
              ],
              [SemanticResourceAttributes.DEPLOYMENT_ENVIRONMENT]:
                process.env.ENV_TIER,
              [SemanticResourceAttributes.CLOUD_ACCOUNT_ID]: accountId,
              [SemanticResourceAttributes.CLOUD_PROVIDER]:
                CloudProviderValues.AWS,
              [SemanticResourceAttributes.CLOUD_REGION]: process.env.AWS_REGION,
              [SemanticResourceAttributes.CLOUD_PLATFORM]:
                CloudPlatformValues.AWS_LAMBDA,
              "cloud.resource_id": functionArn,
              // https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/faas.md
              //TODO infer `faas.trigger`
              //TODO infer `faas.document.*`
              [SemanticResourceAttributes.FAAS_NAME]: context.functionName,
              [SemanticAttributes.FAAS_EXECUTION]: context.awsRequestId,
              [SemanticResourceAttributes.FAAS_ID]: functionArn,
              [SemanticResourceAttributes.FAAS_INSTANCE]: context.logStreamName,
              [SemanticResourceAttributes.FAAS_VERSION]:
                context.functionVersion,
              [SemanticResourceAttributes.FAAS_MAX_MEMORY]:
                Number(context.memoryLimitInMB) * 1024 * 1024,
              [SemanticResourceAttributes.HOST_ARCH]: os.arch(),
              [SemanticResourceAttributes.SERVICE_NAME]:
                process.env.WARRIFY_SERVICE,
              [SemanticResourceAttributes.SERVICE_NAMESPACE]:
                process.env.WARRIFY_NAMESPACE,
            });
          },
          responseHook: (span, { err, res }) => {
            if (err instanceof Error) {
              span.recordException(err);
            }
          },
        }),
        new DnsInstrumentation({}),
        new HttpInstrumentation(),
      ],
    });
  } catch (err: any) {
    logger.error(err);
  }

  const tracer = trace.getTracer(
    process.env.AWS_LAMBDA_FUNCTION_NAME ?? "unknown"
  );

package.json

{
  "name": "@warrify/lambda-middlewares",
  "version": "0.3.10",
  "description": "",
  "keywords": [],
  "homepage": "",
  "license": "ISC",
  "main": "dist/cjs/index.js",
  "module": "dist/esm/index.js",
  "types": "dist/esm/index.d.ts",
  "exports": {
    ".": {
      "import": "./dist/esm/index.js",
      "require": "./dist/cjs/index.js"
    },
    "./*": {
      "import": "./dist/esm/*.js",
      "require": "./dist/cjs/*.js"
    }
  },
  "dependencies": {
    "@aws-sdk/client-dynamodb": "^3.360.0",
    "@aws-sdk/client-secrets-manager": "^3.360.0",
    "@aws-sdk/client-servicediscovery": "^3.360.0",
    "@aws-sdk/client-ssm": "^3.360.0",
    "@aws-sdk/client-sts": "^3.360.0",
    "@aws-sdk/lib-dynamodb": "^3.360.0",
    "@aws-sdk/node-http-handler": "^3.360.0",
    "@aws-sdk/util-dynamodb": "^3.360.0",
    "@middy/cloudwatch-metrics": "^4.5.2",
    "@middy/core": "^4.5.2",
    "@middy/do-not-wait-for-empty-event-loop": "^4.5.2",
    "@middy/error-logger": "^4.5.2",
    "@middy/event-normalizer": "^4.5.2",
    "@middy/http-content-negotiation": "^4.5.2",
    "@middy/http-cors": "^4.5.2",
    "@middy/http-error-handler": "^4.5.2",
    "@middy/http-event-normalizer": "^4.5.2",
    "@middy/http-header-normalizer": "^4.5.2",
    "@middy/http-json-body-parser": "^4.5.2",
    "@middy/http-multipart-body-parser": "^4.5.2",
    "@middy/http-partial-response": "^4.5.2",
    "@middy/http-response-serializer": "^4.5.2",
    "@middy/http-security-headers": "^4.5.2",
    "@middy/http-urlencode-body-parser": "^4.5.2",
    "@middy/http-urlencode-path-parser": "^4.5.2",
    "@middy/input-output-logger": "^4.5.2",
    "@middy/secrets-manager": "^4.5.2",
    "@middy/sqs-partial-batch-failure": "^4.5.2",
    "@middy/ssm": "^4.5.2",
    "@middy/sts": "^4.5.2",
    "@middy/util": "^4.5.2",
    "@middy/validator": "^4.5.2",
    "@opentelemetry/api": "^1.4.1",
    "@opentelemetry/exporter-trace-otlp-http": "^0.40.0",
    "@opentelemetry/id-generator-aws-xray": "^1.1.2",
    "@opentelemetry/instrumentation": "^0.40.0",
    "@opentelemetry/instrumentation-aws-lambda": "^0.35.3",
    "@opentelemetry/instrumentation-aws-sdk": "^0.34.3",
    "@opentelemetry/instrumentation-dns": "^0.31.5",
    "@opentelemetry/instrumentation-http": "^0.40.0",
    "@opentelemetry/propagator-aws-xray": "^1.2.1",
    "@opentelemetry/sdk-node": "^0.40.0",
    "@opentelemetry/sdk-trace-base": "^1.14.0",
    "@opentelemetry/sdk-trace-node": "^1.14.0",
    "@opentelemetry/semantic-conventions": "^1.14.0",
    "@types/aws-lambda": "^8.10.119",
    "ajv": "^8.12.0",
    "ajv-errors": "^3.0.0",
    "ajv-formats": "^2.1.1",
    "ajv-formats-draft2019": "^1.6.1",
    "aws-embedded-metrics": "^4.1.0",
    "aws-lambda": "^1.0.7",
    "aws-xray-sdk-core": "^3.5.0",
    "date-fns": "^2.30.0",
    "dd-trace": "^4.3.0",
    "decimal.js": "^10.4.3",
    "graphql": "^16.7.1",
    "http-status-codes": "^2.2.0",
    "joi": "^17.9.2",
    "lodash": "^4.17.21",
    "src": "link:./",
  }
  "files": [
    "dist/**/*.d.ts",
    "dist/**/*.[tj]s",
    "dist/**/*.[tj]sx",
    "dist/**/*.json"
  ]
}

Relevant log output

Our Lambda functions are instrumented with datadog, hence the stack trace shows datadog files all the way

{
    "errorType": "Runtime.ImportModuleError",
    "errorMessage": "Error: Cannot find module 'is-core-module/package.json'\nRequire stack:\n- /var/task/index.js\n- /opt/nodejs/node_modules/datadog-lambda-js/runtime/user-function.js\n- /opt/nodejs/node_modules/datadog-lambda-js/runtime/index.js",
    "stack": [
        "Runtime.ImportModuleError: Error: Cannot find module 'is-core-module/package.json'",
        "Require stack:",
        "- /var/task/index.js",
        "- /opt/nodejs/node_modules/datadog-lambda-js/runtime/user-function.js",
        "- /opt/nodejs/node_modules/datadog-lambda-js/runtime/index.js",
        "    at ImportModuleError.ExtendedError [as constructor] (/opt/nodejs/node_modules/datadog-lambda-js/runtime/errors.js:113:28)",
        "    at new ImportModuleError (/opt/nodejs/node_modules/datadog-lambda-js/runtime/errors.js:123:42)",
        "    at /opt/nodejs/node_modules/datadog-lambda-js/runtime/user-function.js:273:31",
        "    at step (/opt/nodejs/node_modules/datadog-lambda-js/runtime/user-function.js:43:23)",
        "    at Object.throw (/opt/nodejs/node_modules/datadog-lambda-js/runtime/user-function.js:24:53)",
        "    at rejected (/opt/nodejs/node_modules/datadog-lambda-js/runtime/user-function.js:16:65)"
    ]
}

[deltaepsilon]

We're having the same problem over at https://github.com/highlight/highlight

It just kicked up when we started consuming the CJS version instead of the ESM.

[deltaepsilon]

I made a quick edit to @opentelemetry/instrumentation/build/esm/platform/node/instrumentation.ts:23.

I changed the import to import ImportInTheMiddle from 'import-in-the-middle';.

It worked great.

[dyladan]

@deltaepsilon @matthias-pichler-warrify I think the latest release should have resolved this

[deltaepsilon]

@deltaepsilon @matthias-pichler-warrify I think the latest release should have resolved this

Awesome! I'm working on this now. I'll update and report back.

[deltaepsilon]

@dyladan Any idea on when 0.14.2 will hit NPM?

[dyladan]

Oh I'm sorry I conflated this with a different issue. @trentm do you have any idea what's going on here? I vaguely remember a similar problem with is-core-module in the past.

[trentm]

I'll make some guesses, but I'm not that confident with either esbuild bundling, or with the "build/esm/..." output of OTel packages.

I vaguely remember a similar problem with is-core-module in the past.

is-core-module is imported by resolve which is imported by require-in-the-middle. Ah, the "similar problem in the past" was perhaps with require-in-the-middle, rather than import-in-the-middle. For the earlier case, IIRC, there was this discussion: browserify/resolve#297
(and these two linked issues: elastic/require-in-the-middle#71 #3813)
I think that was resolved when resolve@1.22.2 was marked as the "latest" dist-tag -- rather than the resolve@1.22.3 release which was reaching into the is-core-module/... installation.

I'm not sure that is necessarily the same issue as here. If it is the same issue, then I think you would be broken if npm ls -a resolve shows that resolve@1.22.3 is being used in your install/build.

---

If someone hitting this error created a full repro with the specific commands to run, that would help debug this.

[trentm]

Oh, actually this looks more similar: #3701
(Also any of these matching issues: https://github.com/open-telemetry/opentelemetry-js/issues?q=+%22import+namespace+object%22+)

The fix/workaround then was to:

1. Get a new version of require-in-the-middle that provided a named export for the Hook function -- rather than just being a default export: @opentelemetry/instrumentation crashes when used with typescript esModuleInterop: true #3701 (comment)
2. Then updating the OTel usage to use the named export: https://github.com/open-telemetry/opentelemetry-js/pull/3727/files#diff-c2b4fb5752545bff41e0e0eabf5af35a41ef65c7fb0dec41c258690bdb71e7f9R18

To be honest I get a little lost in the "TypeScript import syntax" -> "commonjs and/or esm built code" -> "use of that code in require() or import ... or import()" -> "possibly bundled by one of many bundlers". However, if this is the same thing (I wasn't able to repro last time), then we'd need/want to do the same changes to import-in-the-middle.

[deltaepsilon]

@trentm I feel your pain regarding ESM/CJS/Whatever-else-they've-come-up-with.

I solved the problem for my local environment by editing the import deep in down in node_modules. I found import * as ImportInTheMiddle from 'import-in-the-middle'; and changed it to import ImportInTheMiddle from 'import-in-the-middle';.

That was all the bundler needed to be happy again.

Of course, the moment I send this to our CI pipeline it'll install the package from scratch.

[matt-kunze-nutrien]

Running into this same issue trying to use opentelemetry in a project built using esbuild. If it helps anyone I ended up working around the issue in a semi-durable way using https://github.com/ds300/patch-package to make the change described above.

Would be awesome to have a "real" fix and not require the hackaround

[jd-carroll]

@pichlermarc - Is there a reason why this:

opentelemetry-js/experimental/packages/opentelemetry-instrumentation/src/platform/node/instrumentation.ts

Line 28 in 20182d8

import * as ImportInTheMiddle from 'import-in-the-middle';

Was not updated to this:

import { ImportInTheMiddle } from 'import-in-the-middle';

In the latest release?

Because at runtime this will absolutely lead to an error being thrown for ESM based applications.

[dyladan]

Already posted in a proposed fix PR but I wanted to put this here too to get more eyes on it:

We're having a hard time validating this fix because we don't have a reproducer. Does anyone have a repro we can use? I'm also not convinced that after we fix this error your problems will be solved. To my knowledge, bundlers work by removing import/require statements and inlining code. Our instrumentations rely on the functioning of those statements so it may turn out that once this problem is fixed the instrumentations may not work anyway.

The obvious thing to do would be to fix the exports in import-in-the-middle and release a V2 that doesn't have invalid exports!

This is 100% what I would prefer. We also talked about changing the web instrumentations to not use the same instrumentation base class as node instrumentations because they work differently (no require/import capture), which would also resolve this issue at least in web.