-
Notifications
You must be signed in to change notification settings - Fork 30.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
RangeError: Invalid string length with large files. #35973
Comments
This is a V8 limitation. I confirm that the maximum number of UTF16 codepoints in a string on 64-bit platforms has passed from |
Yes, the limit is defined here: Lines 3004 to 3007 in eb24573
I don't know why it has this value nor whether this could be increased in the future. |
The limitation is due to pointer compressions, please see: https://chromium-review.googlesource.com/c/v8/v8/+/2030916 |
@aladdin-add Can you update this tests? The test runner wen crazy again, so got the same issues on update and some other failures RangeError: Invalid string length at JSON.stringify (<anonymous>) at outputBlock (D:\kataw15\test\runner\tob.ts:112:8) at Object.updateTob (D:\kataw15\test\runner\tob.ts:23:18) at D:\kataw15\test\cli.ts:32:9 at Generator.next (<anonymous>) at fulfilled (D:\kataw15\node_modules\tslib\tslib.js:114:62)
Consider processing the JSON chunk, by chunk as it comes in, rather than trying to load it all into a string and then parsing it. This will drastically lower the amount of memory needed, and speed up the overall file load time. |
I am also using v8 and got the same error within the code below at:
However, I wounder why this even happens when I cut the data off below 2GB like:
Can you hope me please to overcome this problem, hinting me in of an example ? Thanks!!!
|
Closing as the maximum string length is determined by V8. |
What steps will reproduce the bug?
When i'm trying to read a large Json File (700MB - 26.640.222 lines) using ReadStream i'm getting the error: "RangeError: Invalid string length". Will post the code i'm using to read the file:
How often does it reproduce? Is there a required condition?
Only when the file to be read is very large.
What is the expected behavior?
Not get the error when reading a large file.
What do you see instead?
This is the error that shows to me:
Additional information
The weird thing is, i can read this large file with Node 12.19.0 and Node 13.X versions without problems, only in this new versions 14.X that i'm getting this issue. I'm doing something wrong or there is some workaround this? Because it works nice with almost all the files, only in large files that i'm having this issue.
Thank you.
The text was updated successfully, but these errors were encountered: