Large Codebase Migration Story

[CodeFromAnywhere]

I've succeeded migrating my monorepo from node.js to bun. My codebase currently has over 300 packages and around 100.000 lines of code. I encountered many challenges along the way, so maybe it's useful for someone (and maybe the bun team can use my feedback)

Challenges during migration to bun.js

⛔️ Bun's "node:fs" cp was missing which meant I needed to create this even before migrating.

⛔️ Bun.js seems to have a problem with reading too many files at once, or maybe some other subtle differences in the fs api. Seems to fuck up js memory in some cases. Very weird behavior... Be very careful with this.

⛔️ The server (serverjs.io) package relies on app-module-path which uses Module._nodeModulePaths which does not exist in bun. Therefore, we need to get rid of server.js (we'll replace it anyway with a bun server)

⛔️ Many bun-commands work in vscode terminal but not in regular terminal. There is probably a tiny difference. Not crucial for now, but need to figure this out later

⛔️ Creating an executable for the server that includes all endpoints does not work yet. It now returns `Could not resolve: "@swc/wasm". I opened an issue (#3903), but it seems not to be supported soon. So I need to expose my code as typescript (not a big deal, unless I want my project to be closed source)

⛔️ Upon installing I found that workspace "resolutions" don't work. Because of this, I had duplicate react versions in my node_modules, which causes the famous "invalid hook call" error. I had to dedupe all react and react-dom versions, which took me hours for a project with many packages. Had to install npm for npm ls to find where the problem was because bun doesn't seem to have this functionality.

⛔️ I found that the package https://github.com/vega/ts-json-schema-generator works differently in bun compared to node. It cannot create the named parameters for some functions that were fine before. Apparently declarations is undefined and it crashes in the package. As I rely heavily on this package, I made a PR hot fix, but of course, it means the underlying primitives just work differently, and you probably want to fix that first instead of making compatibility with 100s of packages. At

fs bug

There is probably a logical explanation for this, but I encountered something very strange today with bun. I had an array of strings in a const variable. After logging the result when creating it, it seems fine. After that I called a function in which I didn't provide this variable to the function. This function then, did a lot of (unrelated) file reads using "node:fs.readFile". After this was done, I checked the string array of the const variable again, and it changed to (seemingly random) lines of contents that was in those files.

I suspect that it is some sort of memory bug in bun with fs.readFile. But to be honest, this is a guess as that function did more stuff besides reading the file. It could be another function in bun as well that somehow overwrites memory of an unrelated variable.

Ideas for interchangeability

As there are some huge blockers for me now in Bun, I am forced to return to Node.js.
It was a lot of work to move from node to bun, but the other way around is a lot easier I suspect.

I have one question, maybe someone found a solution. I have a typescript project, so my package.json in 300+ packages in my monorepo looks like this now:

{
  "name": "ai-creative",
  "main": "src/index.ts",
  "peerDependencies": {
    "typescript": "^5.0.0"
  },
  "devDependencies": {
    "bun-types": "latest",
    "@types/web": "^0.0.111"
  },
  "version": "0.0.1",
  "scripts": {
    "css": "tailwindcss -o css.css"
  },
  "license": "ISC",
  "operation": {
    "lastRebuildAt": 1690977759608,
    "isBuildSuccessful": true
  }
}

Before , I had "main": "build/index.js" and "source": "src/index.ts"

I have also changed my tsconfig a lot. It now looks like this:

{
  "compilerOptions": {
    "lib": [
      "ESNext",
      "DOM",
      "DOM.Iterable"
    ],
    "module": "esnext",
    "target": "esnext",
    "moduleResolution": "bundler",
    "resolveJsonModule": true,
    "moduleDetection": "force",
    "allowImportingTsExtensions": true,
    "strict": true,
    "downlevelIteration": true,
    "skipLibCheck": true,
    "jsx": "react-jsx",
    "allowSyntheticDefaultImports": true,
    "forceConsistentCasingInFileNames": true,
    "allowJs": true,
    "noEmit": true,
    "types": [
      "web",
      "bun-types"
    ]
  }
}

I was wondering if I can configure my tsconfig.json and package.json's in a way that I can run my stuff with node as well as bun. Of course there would also be other questions... Can I run the "bun" package in node.js? Can I use bun-types in node? If so, it would yield a much smoother migration, because I can keep trying to make it compatible while still using node.js in the same codebase/branch.

If anyone has experience with this or has any tips/ideas, let me know....

For now, I'm going to probably create a migration script that alters my package.json and tsconfig.json files as desired if I want to switch between bun/node, and also do all other things required.

I'll update this post later with the results.

[Jarred-Sumner]

Really appreciate the feedback!

⛔️ Bun's "node:fs" cp was missing which meant I needed to create this even before migrating.

We will implement this soon

⛔️ Bun.js seems to have a problem with reading too many files at once, or maybe some other subtle differences in the fs api. Seems to fuck up js memory in some cases. Very weird behavior... Be very careful with this.

This might be fixed by #3964

⛔️ Upon installing I found that workspace "resolutions" don't work. Because of this, I had duplicate react versions in my node_modules, which causes the famous "invalid hook call" error. I had to dedupe all react and react-dom versions, which took me hours for a project with many packages. Had to install npm for npm ls to find where the problem was because bun doesn't seem to have this functionality.

Sorry about this. We haven't implemented resolutions yet. It does hoist packages to the root, but that's not good enough

⛔️ Creating an executable for the server that includes all endpoints does not work yet. It now returns `Could not resolve: "@swc/wasm". I opened an issue (#3903), but it seems not to be supported soon. So I need to expose my code as typescript (not a big deal, unless I want my project to be closed source)

This will likely be fixed via default allow-list for"trustedDependencies" which will come in Bun v0.8 or sooner.

⛔️ I found that the package https://github.com/vega/ts-json-schema-generator works differently in bun compared to node. It cannot create the named parameters for some functions that were fine before. Apparently declarations is undefined and it crashes in the package. As I rely heavily on this package, I made a PR hot fix, but of course, it means the underlying primitives just work differently, and you probably want to fix that first instead of making compatibility with 100s of packages. At

This is most likely a CommonJS <> ESM interop issue. If that package uses an older version of tslib it would be fixed in Bun v0.7.2. We will look into it

It was a lot of work to move from node to bun

Sorry about this. We try for it to not be a lot of work but we are still not quite ready yet for production.

There is probably a logical explanation for this, but I encountered something very strange today with bun. I had an array of strings in a const variable. After logging the result when creating it, it seems fine. After that I called a function in which I didn't provide this variable to the function. This function then, did a lot of (unrelated) file reads using "node:fs.readFile". After this was done, I checked the string array of the const variable again, and it changed to (seemingly random) lines of contents that was in those files.

This is likely fixed in #3964. It might be #3931. We added support for async fs functions in v0.7.1 and there were bugs

I was wondering if I can configure my tsconfig.json and package.json's in a way that I can run my stuff with node as well as bun. Of course there would also be other questions... Can I run the "bun" package in node.js? Can I use bun-types in node? If so, it would yield a much smoother migration, because I can keep trying to make it compatible while still using node.js in the same codebase/branch.

@jhmaster2000 is working on making it very easy to use Bun APIs in Node. This will be integrated with bun build

[jhmaster2000]

I'll update this post later with the results.

Something that would be very interesting to hear from this, if you'd be willing to check, would be which Bun APIs you used in the making of the bun version of your codebase. Some real world usage data from a large project such as yours would be useful for considering which APIs to give some extra priority for the Node adaptations.

[CodeFromAnywhere]

Bun API's:

For the server including React SSR

• Serve
• Bun.build
• Bun.FileSystemRouter

For loads of other things:

• Bun.file(xxx).*
• Bun.write

Node API's I heavily rely on:

• fs and fs/promises (also use graceful-fs to circumvent some bugs in fs)
• path
• os
• child_process

Some npm packages I rely on:

• fluent-ffmpeg
• @sendgrid/mail, openai, twilio, stripe, and other API wrappers like that
• jimp
• battery-level, is-charging, desktop-screenshot, node-global-key-listener
• parsers like pdf-parse, csv-parse, vcf, js-yaml
• chokidar (watcher)
• node-cron (cronjobs)
• @swc/core
• ts-json-schema-generator (wanted to try deepkit but didn't work)
• loads of frontend things like popper and react-query. Couldn't run stuff yet with bun, I keep getting the react "invalid hook call" error even after solving all react versions...

There's more...
If you want I can make an overview with what works for me and what doesn't once I encounter it.

I'm now trying to run all bun stuff inside of node. Will that work? E.g. Bun.file, Bun.write, etc.
Most importantly, will it be possible to somehow run the server in the bun way (Serve) while running my code with node? Because I transitioned to that but now I want to make it compatible with both so I can transition fully later more easily.

[jhmaster2000]

Bun.build

Interesting, by this do you mean the actual JS API or the CLI command (bun build)? If it was the JS API I'd be interested in more details on what the use case for that one was. That one is the only really "concerning" one since it likely won't ever be possible to polyfill without a dependency on an actual bun installation to call into, the best case scenario would be a WASM build of bun with just the bundler. Since your usage of it is on the server this shouldn't necessarily be a problem but it is a concern for frontend uses since even the theoretical WASM bundler would probably be a fairly large binary, not ideal for frontends.

I'm now trying to run all bun stuff inside of node. Will that work? E.g. Bun.file, Bun.write, etc.
Most importantly, will it be possible to somehow run the server in the bun way (Serve) while running my code with node? Because I transitioned to that but now I want to make it compatible with both so I can transition fully later more easily.

Currently it will not work no, the plan is to make it work in the future through bun build --target=node which will attach polyfills of Bun APIs such as Bun.file, Bun.write and yes, Bun.serve too, to your bundled code, which will allow it to run in node, but this feature is not yet finished or released, there will be an announcement for it on the Bun release it gets shipped under that you can look out for.

[CodeFromAnywhere]

@jhmaster2000 @Jarred-Sumner I'm considering rewriting and moving away again from Bun.serve. Do you have an ETA on a node-compatible Bun.serve? Because I'd love to be able to wait for that as I really like the API's simplicity. The only thing I've encountered is that it's not possible to do res.write which is the only limitation I've had with it so far (incremental writes for things like streaming tokens of LLM's)

I'm not sure how polyfills work, is it just a javascript implementation of the same thing? Or can you somehow wrap your Zig code? If you need any help on typescript side, I might be able to if you can provide more details.

[jhmaster2000]

No ETA for something guaranteed to be stable, there's quite a lot to do before reaching that stage, polyfills are currently mostly in prototyping stage. Though I plan to look into getting an initial implementation for Bun.serve going either this week or the next.

For most simple APIs they are simply reimplemented in JS (or more specifically TS) since its preferable for tree-shaking the polyfills used, but where necessary or useful the polyfills can also use WASM to integrate with Zig or any other wasm-compilable language (an initial wasm polyfill was in C++ with emscripten for example, but it was changed to a zig one as it was simpler and more efficient). Later on some of the more "magic" APIs that aren't viable to polyfill with JS may have their polyfilling integrated into Bun's bundler and those would be exclusive to it (one example of such unpolyfillable-from-pure-JS api is import.meta.*)

[CodeFromAnywhere]

Okay good to know, thanks.

I was using Bun.build for React SSR on the serverside.

So then this ends my adventure into bun land for now... The reason is, in conclusion, that it is not reliable enough yet for my codebase in production, the biggest reason being that I get segfaults that cannot be debugged by myself.

Critical question: In the priorities you state segfaults are a # 1 priority, but after more than a year of creating that issue they still not solved, while you seem to be working on many other things too. How long before segfaults will be gone?

I'll keep an eye out for bun becoming usable from a node environment. That would be super useful for incrementally adopting bun. If it were possible to run your code from node as well as bun with almost the same package.json and tsconfig settings, it would be most desirable because you can basically keep testing both until bun seems stable enough, and then go there fully. May also provide you insightful information about bun compatibility with tons of packages.

I guess bun is already super useful for many smaller projects, and especially new projects, but unfortunately it seems to be hard to adopt for large codebases like mine.

[CodeFromAnywhere]

UPDATE: NOT Giving up yet 😁

Initially my tsconfig was vastly different compared to what I ended up with after the bun migration. More specifically, I was compiling everything to CommonJS. This fact made it hard to make my codebase compatible with both node.js and bun.

I tried to have bun compile to CommonJS but I got segfaults immideatly. This way I couldnt make the tsconfigs equal.

So I tried the other way...

1. I made all packages in my monorepo modules by changing package.json.type to module
2. I altered all imports everywhere to include the .js extension since this is required for modules
3. Migrated away from __filename and __dirname as well as some other constants that don't exist anymore
4. Migrated away from Bun.file, Bun.write, and Bun.serve, as these aren't available in node yet
5. I made sure that tsconfig is setup to compile the same way as I had setup in my bun project (compile with ESNext)

Now I ended up with there being just a single difference, and that is that in a node monorepo I needed build/index.js as main entry, while in bun it could also be src/index.ts

This is not a big deal, because bun can also run javascript!

After many trials and errors, this is what I came up with.

After migrating all package.jsons to the new entrypoint, I should be able to run my code with node.js as well as with bun.js!

STILL WORK IN PROGRESS. Will update soon

[CodeFromAnywhere]

I ended up removing most references to Bun.* and ensuring that, wherever I still use the bun global, I am importing it like this: import Bun from "bun". This way I can run everything in node as well as in bun. :)

My SSR server currently only runs in Bun (using Bun.build and serve) so I will need to run that from bun. For production I'll try to rewrite something using "node:http".

I hope I can move to bun fully in the coming months, once my main instability problems are solved.

Now it's done! For my typescript monorepo, each tsconfig.json now looks like this:

{
  "compilerOptions": {
    "lib": ["ESNext", "DOM", "DOM.Iterable"],
    "module": "esnext",
    "target": "esnext",
    "moduleResolution": "bundler",
    "resolveJsonModule": true,
    "moduleDetection": "force",
    "allowImportingTsExtensions": false,
    "strict": true,
    "downlevelIteration": true,
    "skipLibCheck": true,
    "jsx": "react-jsx",
    "allowSyntheticDefaultImports": true,
    "forceConsistentCasingInFileNames": true,
    "allowJs": true,
    "noEmit": false,
    "types": ["web", "bun-types"],
    "outDir": "build",
    "sourceMap": true,
    "declarationMap": true,
    "declaration": true,
    "allowArbitraryExtensions": true
  },
  "include": ["src"]
}

This allows me to build using tsc to the build folder. This doesn't impact me running bun stuff, but I can also run the same code with node.js now.

Each package.json looks like this:

{
  "name": "PACKAGE_NAME",
  "main": "build/index.js",
  "type": "module",
  "peerDependencies": {
    "typescript": "^5.0.0"
  },
  "devDependencies": {
    "bun-types": "latest",
    "@types/web": "^0.0.111"
  },
  "dependencies": {},
  "version": "0.0.1",
}

Note the type and main entries.

I created a script that allows me to toggle the main entry between src/index.ts and build/index.js. This allows me to quickly switch between the Bun.js and Node.js runtimes. If I want to run stuff in Node.js (which I do, in production) I need to build all packages first using tsc. In development, I can often simply use tsc.

Maybe this helps more people to migrate big projects. For me this is the best. way currently, since some things are still unstable for me compared to Node.js

[CodeFromAnywhere]

Little update: I have succeeded to make my codebase non-crashing using bun. For some reason, I didn't get any segfaults lately.
I have some more strange things which I'll comment on later, but the good thing is Bun runs react SSR as well as a regular server and I'm loving it!

[mustafa519]

So no more issues since then?