fs.openAsBlob() does not work properly for files > 2GB
#52585
Comments
joelrbrandt
changed the title
fs.openAsBlob does not work properly for files > 2GBfs.openAsBlob() does not work properly for files > 2GB
|
FWIW, this also reproduces on a root@f7a53c9fe57c:/# uname -a
Linux f7a53c9fe57c 6.6.16-linuxkit #1 SMP Fri Feb 16 11:54:02 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux
root@f7a53c9fe57c:/# node -v
v20.12.2
root@f7a53c9fe57c:/# dd if=/dev/random of=random.bin bs=1048576 count=3072
3072+0 records in
3072+0 records out
3221225472 bytes (3.2 GB, 3.0 GiB) copied, 8.45959 s, 381 MB/s
root@f7a53c9fe57c:/# node
Welcome to Node.js v20.12.2.
Type ".help" for more information.
> const b = await fs.openAsBlob("random.bin")
undefined
> b
Blob { size: 3221225472, type: '' }
> b.slice(2**31-1, 2**31)
Blob { size: 0, type: '' }
> b.slice(2**31, 2**31+1)
# node[15]: static void node::Blob::ToSlice(const v8::FunctionCallbackInfo<v8::Value>&) at ../src/node_blob.cc:247
# Assertion failed: args[0]->IsUint32()
----- Native stack trace -----
1: 0xcab09c node::Assert(node::AssertionInfo const&) [node]
2: 0xc65b38 node::Blob::ToSlice(v8::FunctionCallbackInfo<v8::Value> const&) [node]
3: 0xf1f228 v8::internal::FunctionCallbackArguments::Call(v8::internal::CallHandlerInfo) [node]
4: 0xf1f9e8 [node]
5: 0xf1fe00 v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) [node]
6: 0x189c964 [node]
----- JavaScript stack trace -----
1: slice (node:internal/blob:266:21)
2: REPL4:1:3
3: runInThisContext (node:vm:136:12)
4: defaultEval (node:repl:598:22)
5: bound (node:domain:432:15)
6: runBound (node:domain:443:12)
7: onLine (node:repl:927:10)
8: emit (node:events:530:35)
9: emit (node:domain:488:12)
10: [_onLine] (node:internal/readline/interface:416:12)
Aborted
|
VoltrexKeyva
added
the
fs
|
Hi @joelrbrandt , In the implementation of |
|
@kylo5aby thanks for your quick response (and sorry for my slow follow-up).
I'm a bit confused by this part of your response. Where is this hinted? In the node implementation? I don't see any mention of a 32-bit limitation in the File API spec. So, it doesn't seem like the Independent of the issue with Interestingly, if I get a stream of the Blob (with See below where I do the following:
$ dd if=/dev/random of=random.bin bs=1048576 count=5120
5120+0 records in
5120+0 records out
5368709120 bytes transferred in 6.613665 secs (811760063 bytes/sec)
$ openssl dgst random.bin
SHA256(random.bin)= f80fac6d9dc913e33d2c6a69783fd8fc3b4ed18f9c844aca801358e437096e6c
$ node
Welcome to Node.js v20.12.2.
Type ".help" for more information.
> b = await fs.openAsBlob("random.bin")
Blob { size: 1073741824, type: '' }
> b.size
1073741824
> const { createHash } = require("node:crypto")
undefined
> const h = createHash("sha256")
undefined
> const s = b.stream()
undefined
> for await (const a of s) {
... h.update(a);
... }
undefined
> h.digest("hex")
'f80fac6d9dc913e33d2c6a69783fd8fc3b4ed18f9c844aca801358e437096e6c'Finally, here's a codepen that can be used to check that files > 4GB work fine in all major modern browsers (open console to see output): https://codepen.io/joelrbrandt/pen/wvZRXvY |
|
Finally, if
> b = Buffer.alloc(2**33)
Uncaught:
RangeError [ERR_OUT_OF_RANGE]: The value of "size" is out of range. It must be >= 0 && <= 4294967296. Received 8_589_934_592
at Function.alloc (node:buffer:389:3) {
code: 'ERR_OUT_OF_RANGE'
} |
|
Hi @joelrbrandt, Thank you for the information.
There maybe a bug in
|
I'm not sure if this is accurate, either. Trying to construct a new Blob from a set of ArrayBuffers that total a size greater than 4GB also fails: $ node
Welcome to Node.js v18.13.0.
Type ".help" for more information.
> const arrs = []
undefined
> for (let i = 0; i < 5 ; i++) {
... arrs.push(new Uint8Array(2**30))
... }
5
> arrs.reduce((acc, cur) => acc + cur.byteLength, 0)
5368709120
> const b = new Blob(arrs)
Uncaught:
RangeError [ERR_BUFFER_TOO_LARGE]: Cannot create a Buffer larger than 4294967295 bytes
at __node_internal_captureLargerStackTrace (node:internal/errors:491:5)
at new NodeError (node:internal/errors:400:5)
at new Blob (node:internal/blob:165:13) {
code: 'ERR_BUFFER_TOO_LARGE'
}However, this code also works fine in a browser, resulting in a 5GB |
|
You are right. In versions up to v21, on 64-bit systems, the maximum size of a Buffer was 4GB. However, in the version I last used, compiled from the main branch, this limit has been exceeded, that's why I said |
PR-URL: nodejs#52588 Refs: nodejs#52585 Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>
PR-URL: nodejs#52588 Refs: nodejs#52585 Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>
PR-URL: nodejs#52588 Refs: nodejs#52585 Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>
|
It appears to persist this issue for files >4GB in latest Node.js v23.3.0 and v20.18.1: $ dd if=/dev/random of=random.bin bs=1048576 count=5120
$ ls -l random.bin
-rw-r--r-- 1 user user 5368709120 Dec 10 23:39 random.bin
$ node
Welcome to Node.js v23.3.0.
Type ".help" for more information.
> b = await fs.openAsBlob("random.bin")
Blob { size: 1073741824, type: '' }
> b.slice(2**31-1, 2**31)
Blob { size: 0, type: '' }Also: $ node
Welcome to Node.js v23.3.0.
Type ".help" for more information.
> arrs = []; for (let i = 0; i < 5; i++) { arrs.push(new Uint8Array(2**30)); }
5
> b = new Blob(arrs)
Blob { size: 5368709120, type: '' }
> b.slice(2**32-2, 2**32-1) // 4GB-2, 4GB-1
Blob { size: 1, type: '' }
> b.slice(2**32-1, 2**32-0) // 4GB-1, 4GB-0
# node[91790]: static void node::Blob::ToSlice(const FunctionCallbackInfo<v8::Value> &) at ../src/node_blob.cc:264
# Assertion failed: args[1]->IsUint32()
----- Native stack trace -----
1: 0x1026a2ff0 node::Assert(node::AssertionInfo const&) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
2: 0x10449da7c node::Blob::ToSlice(v8::FunctionCallbackInfo<v8::Value> const&) (.cold.1) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
3: 0x10266c86c node::Blob::HasInstance(node::Environment*, v8::Local<v8::Value>) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
4: 0x103452978 Builtins_CallApiCallbackGeneric [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
5: 0x103450838 Builtins_InterpreterEntryTrampoline [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
6: 0x103450838 Builtins_InterpreterEntryTrampoline [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
7: 0x10344e50c Builtins_JSEntryTrampoline [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
8: 0x10344e1b0 Builtins_JSEntry [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
9: 0x102a28760 v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
10: 0x102a28d48 v8::internal::Execution::CallScript(v8::internal::Isolate*, v8::internal::Handle<v8::internal::JSFunction>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
11: 0x1028d3268 v8::Script::Run(v8::Local<v8::Context>, v8::Local<v8::Data>) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
12: 0x102693098 node::contextify::ContextifyScript::EvalMachine(v8::Local<v8::Context>, node::Environment*, long long, bool, bool, bool, v8::MicrotaskQueue*, v8::FunctionCallbackInfo<v8::Value> const&) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
13: 0x1026927ac node::contextify::ContextifyScript::RunInContext(v8::FunctionCallbackInfo<v8::Value> const&) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
14: 0x103452978 Builtins_CallApiCallbackGeneric [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
15: 0x103450838 Builtins_InterpreterEntryTrampoline [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
16: 0x103450838 Builtins_InterpreterEntryTrampoline [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
17: 0x103450838 Builtins_InterpreterEntryTrampoline [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
18: 0x103450838 Builtins_InterpreterEntryTrampoline [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
19: 0x103450838 Builtins_InterpreterEntryTrampoline [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
20: 0x10a006cf4
21: 0x10a04082c
22: 0x103450838 Builtins_InterpreterEntryTrampoline [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
23: 0x103450838 Builtins_InterpreterEntryTrampoline [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
24: 0x10a033c4c
25: 0x10a03db94
26: 0x10a038ffc
27: 0x10a006ba0
28: 0x10a04082c
29: 0x10a038310
30: 0x1034902c4 Builtins_GeneratorPrototypeNext [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
31: 0x10a01d0a0
32: 0x10a006ba0
33: 0x10a04082c
34: 0x10a03a574
35: 0x10a039df8
36: 0x10a03be0c
37: 0x10a0417e4
38: 0x10a00adf0
39: 0x10344e50c Builtins_JSEntryTrampoline [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
40: 0x10344e1b0 Builtins_JSEntry [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
41: 0x102a28760 v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
42: 0x102a280a8 v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
43: 0x1028e4d64 v8::Function::Call(v8::Isolate*, v8::Local<v8::Context>, v8::Local<v8::Value>, int, v8::Local<v8::Value>*) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
44: 0x1025c1c88 node::InternalMakeCallback(node::Environment*, v8::Local<v8::Object>, v8::Local<v8::Object>, v8::Local<v8::Function>, int, v8::Local<v8::Value>*, node::async_context, v8::Local<v8::Value>) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
45: 0x1025d6b34 node::AsyncWrap::MakeCallback(v8::Local<v8::Function>, int, v8::Local<v8::Value>*) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
46: 0x1027aedbc node::StreamBase::CallJSOnreadMethod(long, v8::Local<v8::ArrayBuffer>, unsigned long, node::StreamBase::StreamBaseJSChecks) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
47: 0x1027b0fc0 node::EmitToJSStreamListener::OnStreamRead(long, uv_buf_t const&) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
48: 0x1027b55e0 node::LibuvStreamWrap::OnUvRead(long, uv_buf_t const*) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
49: 0x1027b5ce8 node::LibuvStreamWrap::ReadStart()::$_1::__invoke(uv_stream_s*, long, uv_buf_t const*) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
50: 0x10343a180 uv__stream_io [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
51: 0x103442384 uv__io_poll [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
52: 0x10342ef20 uv_run [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
53: 0x1025c24e4 node::SpinEventLoopInternal(node::Environment*) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
54: 0x1026eb3b0 node::NodeMainInstance::Run(node::ExitCode*, node::Environment*) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
55: 0x1026eb14c node::NodeMainInstance::Run() [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
56: 0x1026615ec node::Start(int, char**) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
57: 0x18d294274 start [/usr/lib/dyld]
----- JavaScript stack trace -----
1: slice (node:internal/blob:271:21)
2: REPL8:1:3
3: runInThisContext (node:vm:137:12)
4: defaultEval (node:repl:597:22)
5: bound (node:domain:433:15)
6: runBound (node:domain:444:12)
7: onLine (node:repl:926:10)
8: emit (node:events:525:35)
9: emit (node:domain:489:12)
10: [_onLine] (node:internal/readline/interface:416:12)
fish: Job 1, 'node' terminated by signal SIGABRT (Abort)Tested on |
Version
v21.7.3 (and v20.12.2)
Platform
Darwin bender.local 23.4.0 Darwin Kernel Version 23.4.0: Fri Mar 15 00:10:42 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T6000 arm64
Subsystem
fs
What steps will reproduce the bug?
Steps to reproduce bug:
dd if=/dev/random of=random.bin bs=1048576 count=3072const b = await fs.openAsBlob("random.bin")b.slice(2**31-2, 2**31-1)<-- worksb.slice(2**31-1, 2**31)<-- returns a slice of zero lengthb.slice(2**31, 2**31+1)<-- crashesFull textual output of doing this given below.
Also, for files greater than 4GB, the size of the blob is wrong. It appears to be exactly
4*(2**30)less than the actual size. For example, for a5*(2**30)byte file, the blob size is reported as1073741824, which is(2**30). And, for a file that is exactly4*(2**30), it reports a size of zero.How often does it reproduce? Is there a required condition?
Always reproduces.
What is the expected behavior? Why is that the expected behavior?
Can operate on the Blob normally.
What do you see instead?
Described above. See output below.
Additional information
The text was updated successfully, but these errors were encountered: