Telemetry breaks embedchain in Nextjs API routes

[Banhawy]

I am using the follwoing:

• Nextjs@14
• typescript@5.1.3

I have an API route that imports embedchain with the following folder structure:

• node_modules/
• app/
  -- api/
  --- pdfchat/
  ---- route.ts <--- embedchain imported here
  -- page.tsx
• next.config.js
• package.json

Here is my route.ts file:

const dotenv = require("dotenv");
dotenv.config();

const { App } = require("embedchain");

export const maxDuration = 120;
export const runtime = "nodejs";

export const dynamic = 'force-dynamic';
dotenv.config();

export async function POST(req: Request) {
  try {
    const {
      messages,
      doc_url,
    } = await req.json()

    const navalChatBot = await App();
    
    await navalChatBot.add('pdf_file', doc_url);
    
    const userMessage = messages[messages.length - 1].content;
    
    const result = await navalChatBot.query(userMessage);

    return result


  } catch (error) {
    console.error(error);
  }
}

However, when this route receives a request and invokes the EmbedChainApp it throws a runtime error as follows:

 ⨯ unhandledRejection: Error: ENOENT: no such file or directory, open '/Users/adham/Developer/rna/chat-etf-bot/.next/server/package.json'
    at Object.openSync (node:fs:603:3)
    at Object.readFileSync (node:fs:471:35)
    at EmbedChainApp.eval (webpack-internal:///(rsc)/./node_modules/.pnpm/embedchain@0.0.8_@aws-sdk+client-dynamodb@3.414.0_@aws-sdk+client-s3@3.456.0_@pinecone-databa_q3mszccedqc6bzmo3cdgeljek4/node_modules/embedchain/dist/embedchain.js:308:47)
    at Generator.next (<anonymous>)
    at eval (webpack-internal:///(rsc)/./node_modules/.pnpm/embedchain@0.0.8_@aws-sdk+client-dynamodb@3.414.0_@aws-sdk+client-s3@3.456.0_@pinecone-databa_q3mszccedqc6bzmo3cdgeljek4/node_modules/embedchain/dist/embedchain.js:59:71)
    at new Promise (<anonymous>)
    at __awaiter (webpack-internal:///(rsc)/./node_modules/.pnpm/embedchain@0.0.8_@aws-sdk+client-dynamodb@3.414.0_@aws-sdk+client-s3@3.456.0_@pinecone-databa_q3mszccedqc6bzmo3cdgeljek4/node_modules/embedchain/dist/embedchain.js:41:12)
    at EmbedChainApp.sendTelemetryEvent (webpack-internal:///(rsc)/./node_modules/.pnpm/embedchain@0.0.8_@aws-sdk+client-dynamodb@3.414.0_@aws-sdk+client-s3@3.456.0_@pinecone-databa_q3mszccedqc6bzmo3cdgeljek4/node_modules/embedchain/dist/embedchain.js:301:16)
    at new EmbedChain (webpack-internal:///(rsc)/./node_modules/.pnpm/embedchain@0.0.8_@aws-sdk+client-dynamodb@3.414.0_@aws-sdk+client-s3@3.456.0_@pinecone-databa_q3mszccedqc6bzmo3cdgeljek4/node_modules/embedchain/dist/embedchain.js:93:14)
    at new EmbedChainApp (webpack-internal:///(rsc)/./node_modules/.pnpm/embedchain@0.0.8_@aws-sdk+client-dynamodb@3.414.0_@aws-sdk+client-s3@3.456.0_@pinecone-databa_q3mszccedqc6bzmo3cdgeljek4/node_modules/embedchain/dist/embedchain.js:344:1)
    at eval (webpack-internal:///(rsc)/./node_modules/.pnpm/embedchain@0.0.8_@aws-sdk+client-dynamodb@3.414.0_@aws-sdk+client-s3@3.456.0_@pinecone-databa_q3mszccedqc6bzmo3cdgeljek4/node_modules/embedchain/dist/index.js:35:21)
    at Generator.next (<anonymous>)
    at eval (webpack-internal:///(rsc)/./node_modules/.pnpm/embedchain@0.0.8_@aws-sdk+client-dynamodb@3.414.0_@aws-sdk+client-s3@3.456.0_@pinecone-databa_q3mszccedqc6bzmo3cdgeljek4/node_modules/embedchain/dist/index.js:26:71)
    at new Promise (<anonymous>)
    at __awaiter (webpack-internal:///(rsc)/./node_modules/.pnpm/embedchain@0.0.8_@aws-sdk+client-dynamodb@3.414.0_@aws-sdk+client-s3@3.456.0_@pinecone-databa_q3mszccedqc6bzmo3cdgeljek4/node_modules/embedchain/dist/index.js:8:12)
    at App (webpack-internal:///(rsc)/./node_modules/.pnpm/embedchain@0.0.8_@aws-sdk+client-dynamodb@3.414.0_@aws-sdk+client-s3@3.456.0_@pinecone-databa_q3mszccedqc6bzmo3cdgeljek4/node_modules/embedchain/dist/index.js:34:17)
    at doc_embedder (webpack-internal:///(rsc)/./app/api/chat/related-docs/doc_embedder.js:6:25)
    at POST (webpack-internal:///(rsc)/./app/api/chat/related-docs/route.ts:25:82)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /Users/adham/Developer/rna/chat-etf-bot/node_modules/.pnpm/next@14.0.4-canary.17_@babel+core@7.23.3_react-dom@18.2.0_react@18.2.0_sass@1.69.5/node_modules/next/dist/compiled/next-server/app-route.runtime.dev.js:6:62623 {
  errno: -2,
  syscall: 'open',
  code: 'ENOENT',
  path: '/Users/adham/Developer/rna/chat-etf-bot/.next/server/package.json'
}
Error: Setting up fake worker failed: "Cannot find module './pdf.worker.js'
Require stack:
- /Users/adham/Developer/rna/chat-etf-bot/.next/server/vendor-chunks/pdfjs-dist@3.11.174.js".
    at eval (webpack-internal:///(rsc)/./node_modules/.pnpm/pdfjs-dist@3.11.174/node_modules/pdfjs-dist/build/pdf.js:2116:58)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

Upon a deeper look, I noticed that the sendTelemetryEvent function is trying to read the contents of the package.json of the embedchain module. However the way the path to the package.json file is declared with __dirname causes a problem. https://github.com/embedchain/embedchainjs/blob/a833b3988a4bb5b5d22cfd96851572f3cc26de24/embedchain/embedchain.ts#L265)

The __dirname variable in this case points to the root directory of the Next.js project, not the directory of the npm module. This means that the package.json file of the npm module won't be found, and fs.readFileSync will throw that error.

Is it possible to use absolute path instead here? Right now this library can't be used in Nextjs APIs 😞