[CI] browser-test are broken

[Flarna]

see e.g.
https://github.com/open-telemetry/opentelemetry-js-contrib/actions/runs/3312465704/jobs/5469167967
https://github.com/open-telemetry/opentelemetry-js-contrib/actions/runs/3312081248/jobs/5468319602

> nyc karma start --single-run

ℹ ｢wdm｣: Hash: 9d2c943b68425fd58dd0
Version: webpack 4.46.0
Time: 35ms
Built at: 10/24/2022 11:26:46 AM
ℹ ｢wdm｣: Compiled successfully.
ℹ ｢wdm｣: Compiling...
----------|---------|----------|---------|---------|-------------------
File      | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
----------|---------|----------|---------|---------|-------------------
All files |       0 |        0 |       0 |       0 |                   
----------|---------|----------|---------|---------|-------------------
lerna ERR! npm run test:browser stderr:

<--- Last few GCs --->

[7536:0x4a892d0]    38085 ms: Scavenge (reduce) 2043.9 (2048.3) -> 2043.4 (2049.5) MB, 19.4 / 0.0 ms  (average mu = 0.271, current mu = 0.240) allocation failure 
[7536:0x4a892d0]    38110 ms: Scavenge (reduce) 2044.1 (2048.5) -> 2043.5 (2049.8) MB, 24.4 / 0.0 ms  (average mu = 0.271, current mu = 0.240) allocation failure 
[7536:0x4a892d0]    38125 ms: Scavenge (reduce) 2044.4 (2051.8) -> 2043.8 (2052.0) MB, 14.9 / 0.0 ms  (average mu = 0.271, current mu = 0.240) allocation failure 


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0xa3ac10 node::Abort() [node]
 2: 0x970199 node::FatalError(char const*, char const*) [node]
 3: 0xbba5ce v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xbba947 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xd76b65  [node]
 6: 0xd776ef  [node]
 7: 0xd8552b v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 8: 0xd890ec v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
 9: 0xd577cb v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
10: 0x109fee0 v8::internal::Runtime_AllocateInOldGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
11: 0x1448f99  [node]
Aborted (core dumped)
npm ERR! code ELIFECYCLE
npm ERR! errno 134
npm ERR! @opentelemetry/instrumentation-user-interaction@0.31.0 test:browser: `nyc karma start --single-run`
npm ERR! Exit status 134
npm ERR! 
npm ERR! Failed at the @opentelemetry/instrumentation-user-interaction@0.31.0 test:browser script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/runner/.npm/_logs/2022-10-24T11_28_03_346Z-debug.log
lerna ERR! npm run test:browser exited 134 in '@opentelemetry/instrumentation-user-interaction'
lerna WARN complete Waiting for 1 child process to exit. CTRL-C to exit immediately.
npm ERR! code ELIFECYCLE
npm ERR! errno 134
npm ERR! opentelemetry-contrib@0.1.0 test:browser: `lerna run test:browser --concurrency 1`
npm ERR! Exit status 134
npm ERR! 
npm ERR! Failed at the opentelemetry-contrib@0.1.0 test:browser script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/runner/.npm/_logs/2022-10-24T11_28_03_384Z-debug.log
Error: Process completed with exit code 134.

Not sure if more memory would help here.

[Flarna]

on my local machine test passes but needs >3GB or RAM. so likely more memory would help.
But not sure if it is correct if this test need such a lot memory.

[blumamir]

I see other people had this issue, but no clue how they ended up resolving it
codymikol/karma-webpack#382
karma-runner/karma#3031

It looks like the webpack invocation by karma is the problematic step.
@MSNev maybe you can spot what's wrong?

[MSNev]

I don't see anything obvious, but it's probably a version dependency issue... I'm trying to repo locally and see what I can find.
But I have had issues with specific versions of karma and webpack previously

[dyladan]

on my local machine test passes but needs >3GB or RAM. so likely more memory would help. But not sure if it is correct if this test need such a lot memory.

our test suite is getting a little out of hand...

[Flarna]

The brower tests use quite some tools and start chrome. That's massive compared to simply running a node process for node.js tests. With that in mind 3-4GB doesn't seem that high.

[Flarna]

seems like it recovered, latest run in #1246 was green.

[MSNev]

I've managed to repo locally and I have the linked PR to fix (hopefully) on GitHub action (marked as draft until it passes)

[MSNev]

Test as now passing so I've marked the PR as ready for review. I've also checked again locally (based on the latest #1264 pass being green) and I can still repo locally, without the changes I have in the PR.

[blumamir]

Some more input

1. I run the tests for long-task instrumentation and found that they pass even for 256MB. A similar run on user-interactions needs >2GB of memory.
2. It looks to me like the step that is failing is running webpack. the last thing it prints before crashing is "Compiling...", when it passes, after this text there is a lot of webpack logging which are missing in user-interactions.
3. I tried to run webpack manually via the webpack-cli with the same config and it seems to fail as well for 2GB. So I guess the issue is not related to karma but to webpack or it's configuration.

➜  opentelemetry-instrumentation-user-interaction git:(master) ✗ export NODE_OPTIONS="--max-old-space-size=2048"
➜  opentelemetry-instrumentation-user-interaction git:(master) ✗ npm exec webpack-cli build                     

<--- Last few GCs --->

[82288:0x7fe108a00000]    19623 ms: Scavenge 2035.1 (2075.6) -> 2031.5 (2076.1) MB, 3.3 / 0.0 ms  (average mu = 0.267, current mu = 0.195) allocation failure 
[82288:0x7fe108a00000]    19628 ms: Scavenge 2035.9 (2076.4) -> 2032.2 (2076.9) MB, 3.2 / 0.0 ms  (average mu = 0.267, current mu = 0.195) allocation failure 
[82288:0x7fe108a00000]    19633 ms: Scavenge 2036.8 (2077.4) -> 2033.2 (2077.9) MB, 3.1 / 0.0 ms  (average mu = 0.267, current mu = 0.195) allocation failure 


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x10dc17ef5 node::Abort() (.cold.1) [/usr/local/bin/node]
 2: 0x10c90b869 node::Abort() [/usr/local/bin/node]
 3: 0x10c90b9df node::OnFatalError(char const*, char const*) [/usr/local/bin/node]
 4: 0x10ca8df07 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
 5: 0x10ca8dea3 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
 6: 0x10cc2f585 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/usr/local/bin/node]
 7: 0x10cc335cd v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [/usr/local/bin/node]
 8: 0x10cc2fead v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/usr/local/bin/node]
 9: 0x10cc2d3cd v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/local/bin/node]
10: 0x10cc2c0e8 v8::internal::Heap::HandleGCRequest() [/usr/local/bin/node]
11: 0x10cbd8541 v8::internal::StackGuard::HandleInterrupts() [/usr/local/bin/node]
12: 0x10cfbecd8 v8::internal::Runtime_StackGuard(int, unsigned long*, v8::internal::Isolate*) [/usr/local/bin/node]
13: 0x10d368f99 Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit [/usr/local/bin/node]
[1]    82287 abort      npm exec webpack-cli build

webpack.config.js

module.exports = {
    mode: 'development',
    target: 'web',
    output: { filename: 'bundle.js' },
    resolve: { extensions: [ '.ts', '.js', '.tsx' ] },
    devtool: 'inline-source-map',
    module: {
      rules: [
        { test: /\.tsx?$/, use: 'ts-loader' },
        {
          enforce: 'post',
          exclude: /(node_modules|\.test\.[tj]sx?$)/,
          test: /\.ts$/,
          use: {
            loader: '@jsdevtools/coverage-istanbul-loader',
            options: { esModules: true }
          }
        },
        {
          parser: {
            node: {
              assert: true,
              global: true,
              Buffer: false,
              __dirname: false,
              __filename: false,
              buffer: false,
              child_process: false,
              cluster: false,
              console: false,
              constants: false,
              crypto: false,
              dgram: false,
              dns: false,
              domain: false,
              events: false,
              fs: false,
              http: false,
              https: false,
              module: false,
              net: false,
              os: false,
              path: false,
              process: false,
              punycode: false,
              querystring: false,
              readline: false,
              repl: false,
              setImmediate: false,
              stream: false,
              string_decoder: false,
              sys: false,
              timers: false,
              tls: false,
              tty: false,
              url: false,
              util: false,
              vm: false,
              zlib: false
            }
          }
        }
      ]
    }
};

[blumamir]

Another thing I just discovered - if I remove the @jsdevtools/coverage-istanbul-loader rule from webpack config, it now works for user-interactions even for 256MB.
So I guess the code coverage is the issue here. Does it make sense that code coverage takes so much memory?

  module: {
    rules: [
      { test: /\.tsx?$/, use: 'ts-loader' },
      // {
      //   enforce: 'post',
      //   exclude: /(node_modules|\.test\.[tj]sx?$)/,
      //   test: /\.ts$/,
      //   use: {
      //     loader: '@jsdevtools/coverage-istanbul-loader',
      //     options: { esModules: true }
      //   }
      // },
      // This setting configures Node polyfills for the browser that will be
      // added to the webpack bundle for Karma tests.
      { parser: { node: webpackNodePolyfills } }
    ]
  }

[blumamir]

My quest continues...

As I wrote before, the issue comes from @jsdevtools/coverage-istanbul-loader Which is a dependency in our repo:

{
  "name": "@opentelemetry/instrumentation-user-interaction",
  "devDependencies": {
    "@jsdevtools/coverage-istanbul-loader": "3.0.5",
  }
}

If I look at the dependency of this package, I find that it has a caret dependency on

    "convert-source-map": "^1.7.0",

This package released a new version 1.9.0 (which is picked up by this caret range) just when our CI issues started (10 October)

I guess this new convert-source-map package now consumes more memory than it used to?

There is no CHANGELOG for it, but these are the things that changed in code.

Looks like it added some decoding which might be to related to memory consumption:

var decodeBase64;
if (typeof Buffer !== 'undefined') {
  if (typeof Buffer.from === 'function') {
    decodeBase64 = decodeBase64WithBufferFrom;
  } else {
    decodeBase64 = decodeBase64WithNewBuffer;
  }
} else {
  decodeBase64 = decodeBase64WithAtob;
}

function decodeBase64WithBufferFrom(base64) {
  return Buffer.from(base64, 'base64').toString();
}

function decodeBase64WithNewBuffer(base64) {
  if (typeof value === 'number') {
    throw new TypeError('The value to decode must not be of type number.');
  }
  return new Buffer(base64, 'base64').toString();
}

function decodeBase64WithAtob(base64) {
  return decodeURIComponent(escape(atob(base64)));
}

[blumamir]

fixed in #1264