AWS Lambda Serverless Framework v4 Sentry Errors with nodeProfilingIntegration

[k-lombard]

Is there an existing issue for this?

• I have checked for existing issues https://github.com/getsentry/sentry-javascript/issues
• I have reviewed the documentation https://docs.sentry.io/
• I am using the latest SDK release https://github.com/getsentry/sentry-javascript/releases

How do you use Sentry?

Sentry Saas (sentry.io)

Which SDK are you using?

@sentry/aws-serverless

SDK Version

8.27.0

Framework Version

nodejs20.x

Link to Sentry event

No response

Reproduction Example/SDK Setup

Using the Serverless Framework v4 with AWS Lambdas.

import * as Sentry from '@sentry/aws-serverless'
import { nodeProfilingIntegration } from '@sentry/profiling-node'
Sentry.init({
  dsn: process.env.SENTRY_DSN,
  // Add Tracing by setting tracesSampleRate and adding integration
  // Set tracesSampleRate to 1.0 to capture 100% of transactions
  // We recommend adjusting this value in production
  maxBreadcrumbs: 50,
  debug: true,
  integrations: [nodeProfilingIntegration()],
  tracesSampleRate: 1.0,
  initialScope: {
    tags: { service: 'service-name' }
  },
  profilesSampleRate: 1.0
})

This leads to this error and my lambdas don't work (only when I include nodeProfilingIntegration)

2024-10-14T19:24:42.682Z	undefined	ERROR	Uncaught Exception 	{"errorType":"TypeError","errorMessage":"The \"path\" argument must be of type string or an instance of URL. Received undefined","code":"ERR_INVALID_ARG_TYPE","stack":["TypeError [ERR_INVALID_ARG_TYPE]: The \"path\" argument must be of type string or an instance of URL. Received undefined","    at Object.fileURLToPath (node:internal/url:1461:11)","    at Object.<anonymous> (/node_modules/@sentry/aws-serverless/src/sdk.ts:378:50)","    at Module._compile (node:internal/modules/cjs/loader:1469:14)","    at Module._extensions..js (node:internal/modules/cjs/loader:1548:10)","    at Module.load (node:internal/modules/cjs/loader:1288:32)","    at Module._load (node:internal/modules/cjs/loader:1104:12)","    at Module.require (node:internal/modules/cjs/loader:1311:19)","    at require (node:internal/modules/helpers:179:18)","    at _tryRequireFile (file:///var/runtime/index.mjs:1002:37)","    at _tryRequire (file:///var/runtime/index.mjs:1052:25)"]}

If I remove nodeProfilingIntegration, my lambdas at least function, but I see TONS of "Sentry Logger [error]" everywhere in my lambda logs.

Those errors look like this:

2024-10-14T18:27:27.208Z	66c09dda-0a5d-44f3-898f-b69fe37785c8	ERROR	Sentry Logger [error]: Failed to read file: /src/handler.ts. Error: Error: ENOENT: no such file or directory, open '/src/handler.ts'

2024-10-14T18:27:27.208Z	66c09dda-0a5d-44f3-898f-b69fe37785c8	ERROR	Sentry Logger [error]: Failed to read file: /node_modules/@sentry/aws-serverless/src/sdk.ts. Error: Error: ENOENT: no such file or directory, open '/node_modules/@sentry/aws-serverless/src/sdk.ts'

Steps to Reproduce

Using Serverless Framework v4 with esbuild to write and deploy my AWS Lambda functions.

Here's my package.json:

{
  "name": "service-name",
  "private": true,
  "version": "1.0.0",
  "description": "",
  "main": "src/handler.ts",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "i": "yarn install --force"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "@aws-sdk/client-ssm": "^3.645.0",
    "@sentry/aws-serverless": "^8.27.0",
    "@sentry/profiling-node": "^8.27.0",
    "auth-utils": "./common/auth-utils",
    "http-utils": "./common/http-utils",
    "joi": "^17.13.3",
    "axios": "^1.7.7",
    "pg": "^8.12.0",
    "pg-hstore": "^2.3.4",
    "plaid": "^26.0.0",
    "plaid-utils": "./common/plaid-utils",
    "rds-utils": "./common/rds-utils",
    "sequelize": "^6.37.3",
    "temporal-utils": "./common/temporal-utils",
    "types-and-constants": "./common/types-and-constants"
  },
  "devDependencies": {
    "@types/node": "^14.14.31",
    "serverless-domain-manager": "^7.3.8"
  }
}

The lambdas are written in typescript and bundled with the built-in esbuild with serverless v4 in sls deploy

Expected Result

Expected result is for node profiling to work and to not have tons of random sentry errors everywhere in my lambda logs.

Actual Result

Logs attached above.

[Jimmy89]

This looks very much related to the recent added documentation about the node profiler using process.env variables to resolve the profiler path (My issue: getsentry/sentry-docs#11128).

I use webpack, so the comments below will be different for esbuild, but hopefully it contains all information to adjust your own config.

I resolved this by copying the sentry_cpu_profiler* files to the dist folder (webpack output) and setting the env variables within the config.

      "env.SENTRY_PROFILER_BINARY_PATH": "false",
      'env.SENTRY_PROFILER_BINARY_DIR': "'./sentry/'", <--- the output folder within the dist folder, containing the sentry cpu profiles files.

As bonus I trim the copied cpu profiles files by running (change to your lambda target):

node_modules/.bin/sentry-prune-profiler-binaries --target_dir_path=./dist/lambda/sentry --target_platform=linux --target_node=20 --target_stdlib=glibc --target_arch=x64

My webpack copy code snippet:

{
              from: path.resolve(__dirname, './node_modules/@sentry/profiling-node/lib/sentry_cpu_profiler-*.node'),
              to: "./sentry/[name][ext]",
            },

Be aware that you keep other process.env references intact when building with esbuild, as lambda plugins might read the lambda environment variables for specific code execution

[andreiborza]

Hello @k-lombard, thanks for writing in. As mentioned in a related comment here, @sentry/profling-node can't simply be bundled as it has binary files.

There are a couple of approaches here you could take. One is the create a dedicated @sentry/profiling-node layer as mentioned in my linked comment. The other is to adjust your build system to properly set up the binary discovery as @Jimmy89 mentions here.

Thank you so much for chiming in and helping out @Jimmy89!

cc @JonasBa
I have creating a dedicated layer for profling-node on my plate, but not sure when I can get around to that as it's currently lower priority. Maybe if you have some spare cycles you could look into that: #13049.

[k-lombard]

@andreiborza @Jimmy89 Even if I turn off nodeProfiling in my sentry_init.js, I still get errors of this type though:

2024-10-14T18:27:27.208Z	66c09dda-0a5d-44f3-898f-b69fe37785c8	ERROR	Sentry Logger [error]: Failed to read file: /src/handler.ts. Error: Error: ENOENT: no such file or directory, open '/src/handler.ts'

Are you aware of how to fix the ENOENT errors? From what I read I assumed it was related to sourcemaps.

Also, @Jimmy89 is the method via which you copy the profiler files to the dist a script in your CI/CD pipeline? Or how are you handling that?

Furthermore, the error events that are sent to sentry currently do not have a corresponding stack trace for my lamdba functions.

And all my lambas are wrapped in wrapHandler: export const patchTruckApi = Sentry.wrapHandler(async (event: any): Promise<HttpResponse> => {

I was able to setup sourcemaps with this esbuild config. I tried doing what you mentioned for esbuild, but couldnt seem to get it working. Im still getting the same error.

esbuild.config.js

const env = require('esbuild-plugin-env')
const { sentryEsbuildPlugin } = require('@sentry/esbuild-plugin')
const { copy } = require('esbuild-plugin-copy')

module.exports = (serverless) => {
  return {
    bundle: true,
    minify: true,
    sourcemap: true,
    exclude: ['@aws-sdk/*'],
    plugins: [
      env(),
      sentryEsbuildPlugin({
        authToken: serverless.service.provider.environment.SENTRY_AUTH_TOKEN,
        org: 'my-org',
        project: 'node-awslambda'
      }),
      copy({
        assets: [
          {
            from: ['./node_modules/@sentry/profiling-node/lib/sentry_cpu_profiler-*.node'],
            to: ['./sentry']
          }
        ]
      })
    ],
    define: {
      'process.env.SENTRY_PROFILER_BINARY_PATH': 'false',
      'process.env.SENTRY_PROFILER_BINARY_DIR': '"/var/task/sentry/"'
    }
  }
}

sentry_init.js:

import * as Sentry from '@sentry/aws-serverless'
import { nodeProfilingIntegration } from '@sentry/profiling-node'

Sentry.init({
  dsn: process.env.SENTRY_DSN,
  // Add Tracing by setting tracesSampleRate and adding integration
  // Set tracesSampleRate to 1.0 to capture 100% of transactions
  // We recommend adjusting this value in production
  maxBreadcrumbs: 50,
  debug: true,
  tracesSampleRate: 1.0,
  integrations: [
    nodeProfilingIntegration({
      profilerBinaryPath: '/var/task/sentry/sentry_cpu_profiler-linux-x64-gnu.node'
    })
  ],
  initialScope: {
    tags: { service: 'assets' }
  },
  environment: process.env.stage,
  profilesSampleRate: 1.0
})

[andreiborza]

@k-lombard can you try turning debug on in the sentryEsbuildPlugin and the sdk init and paste some logs? A reproduction repo would also help to investigate this further.

[k-lombard]

@andreiborza I dont know if Ill be able to make a reproduction repo but I have turned on debug on both. Where should I be looking for sentry-esbuild-plugin logs? When I deploy my lambda I see some logs for the plugin but no errors/warnings or anything.

[chargome]

@k-lombard maybe you could spot anything with sls deploy --verbose

[Jimmy89]

@k-lombard in our pipeline we do the full yarn install, webpack build and uploading the generated dist folder (containing all files) as zip file to s3 (we use Pulumi to update the function).

Looking at your error messages, I have the feeling that esbuild has not resolved dynamic imports, transpiled files should for example not require .ts files. The error logs indicate that they are still there. There are some dynamicRequire and await import statements within the sentry code. I found that esbuild has some issues with dynamic imports

[k-lombard]

@Jimmy89 hmmm yeah I probably need to find someone who knows how to fix these issues for Serverless v4 Framework with the built-in esbuild. I tried a ton of different fixes over the last week with nothing working. Hopefully this becomes somewhat of a priority for the Sentry team soon. If not, my team might have to migrate off of sentry.

@chargome sls deploy --verbose didn't provide any more insights unfortunately.

[s1gr1d]

Hello, this is hard to debug without a minimal reproduction example. Are you able to provide one?

And can you share the logs you are getting?

[JonasBa]

@k-lombard the issue in esbuild with dynamic imports shouldn't cause an issue here. What happens is that if esbuild sees a dynamic import, it wont invoke the loader and copy the binary as it cant eagerly evaluate the expression.

Have you tried marking the package as external like @andreiborza recommended here? If you are using esbuild, you will have to also make sure you exclude @sentry/profiling-node from the bundle by adding external: ['@sentry/profiling-node'], to your esbuild configuration (esbuild documentation).

[getsantry]

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you remove the label Waiting for: Community, I will leave it alone ... forever!

---

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀