1BRC in Node.js 23 Sec and in Bun 14 Sec (but there is a catch with the latter 😁)

[Edgar-P-yan]

Repositories

The old repo for both Node.js and Bun that I created in my own account: https://github.com/Edgar-P-yan/1brc-nodejs-bun
UPDATE: The new and active repositories that I created in the @1brc organization, and where you can submit your own solutions:

• 1BRC for Bun: https://github.com/1brc/bun
• 1BRC for Node.js: https://github.com/1brc/nodejs

Straight to the code:

🐢 Nodejs version.
🍞 Bun version.

Details:

• Machine: MacBook Pro M1 Max 32GB.
• Best results I could get with Node.js is 23s, while with Bun "potentially" around 14 seconds (see the "Problems with Bun").
• Utilizes all of the cores of the system (10 on mine) via worker threads (except for Bun, see below).
• A custom input-specific and quite fast float point to integer parser (minus around 10 seconds compared to the parseFloat, parseInt and Numer()).
• Byte-by-byte processing of the whole file.
• Statically typed code that gets JITed very efficiently.

What should be improved:

• Custom hashmap with 2-byte hashes. Right now I just use the builtin Map, which is too general purpose for this task, hence quite slow, around 30% of spent time is on the Map.
• Set higher watermarks for the GC, it might save some milliseconds too.
• Do fewer allocations. Right now for each station name in each row a new string gets allocated.

Problems with Bun

I noted that it "potentially" runs in 14s. The problem with Bun is that i caught a bug (I reported it here) where i couldnt read the file at a random position if that random position is greater than around 65k (the size of the buffer). But a tenth of the measurements.txt file got processed in 13s on just one core, so if Bun fixes it's seek-and-read feature then on my 10 core machine it'll probably fit in 14s range accounting for the final results formation, thus being faster by 40% then nodejs.

Try out:

# First generate the 13GB `measurements.txt` file. For this step you'll need the Java installed:
./create_measurements.sh 1000000000

# Node.js 23s on my machine:
sh calculate_average_Edgar-P-Yan_nodejs.sh measurements.txt

# Bun, for now only single threaded due to the bug in Bun's stdlib, so takes around 120s on my machine:
sh calculate_average_Edgar-P-Yan_bun.sh measurements.txt

[Edgar-P-yan]

I was also considering to use mmap, but node.js does not provide any builtin API for that. There are some third party native addons that provide mmap support, but the purpose of this challenge is rather to optimize the JS code itself.