Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

memory crash with iterate() after 10M records #280

Closed
kekscom opened this issue Jul 6, 2019 · 3 comments
Closed

memory crash with iterate() after 10M records #280

kekscom opened this issue Jul 6, 2019 · 3 comments

Comments

@kekscom
Copy link

kekscom commented Jul 6, 2019

We're experiencing memory crashes when iterate() reaches about 10M results.
No matter what complexity of the query or code in loop body is.

const db = new SQLite('data.db', {readonly: true});
const select = db.prepare(`select rowid FROM buildings`);
for (const row of select.iterate()) {}
<--- Last few GCs --->

[20359:0x3165040]   117429 ms: Scavenge 1396.9 (1423.2) -> 1396.1 (1424.2) MB, 2.1 / 0.0 ms  (average mu = 0.141, current mu = 0.097) allocation failure 
[20359:0x3165040]   117437 ms: Scavenge 1397.1 (1424.2) -> 1396.3 (1424.7) MB, 1.7 / 0.0 ms  (average mu = 0.141, current mu = 0.097) allocation failure 
[20359:0x3165040]   117461 ms: Scavenge 1397.2 (1424.7) -> 1396.4 (1425.2) MB, 2.3 / 0.0 ms  (average mu = 0.141, current mu = 0.097) allocation failure 

<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0xd7a19e5c01d]
    1: StubFrame [pc: 0xd7a19e5d5a6]
Security context: 0x20404611e549 <JSObject>
    2: log [0x2291bbbf4359] [console.js:~196] [pc=0xd7a19ef91f8](this=0x18a276eb32b9 <Console map = 0x10f7acb4b421>)
    3: arguments adaptor frame: 1->0
    4: InternalFrame [pc: 0xd7a19e0ee55]
    5: EntryFrame [pc: 0xd7a19e09521]
    6: ExitFrame [pc: 0xd7a19e5c01d]
    7: /* anonymous */ [0x18a276eb3479] [/srv/server/e...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x8c02c0 node::Abort() [node]
 2: 0x8c030c  [node]
 3: 0xad15de v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xad1814 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xebe752  [node]
 6: 0xebe858 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [node]
 7: 0xeca982 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]
 8: 0xecb2b4 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 9: 0xecdf21 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [node]
10: 0xe97444 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [node]
11: 0x1136d5e v8::internal::Runtime_AllocateInNewSpace(int, v8::internal::Object**, v8::internal::Isolate*) [node]
12: 0xd7a19e5c01d 

Node v10.9.0
No special settings nor parameters
e.g. no. of lines until crash: 10109787, 10053311, 10137944 ...

@kekscom
Copy link
Author

kekscom commented Jul 7, 2019

It seems number of result records is the issue.
We're querying 350+M records without WHERE / LIMIT.

@kekscom
Copy link
Author

kekscom commented Jul 7, 2019

Can be tested easily on a large DB. LIMIT helps to get closer to the spot. 8M records here.
Script even completes but after that JS Stacktrace shows up.

SELECT rowid
FROM building
LIMIT 8000000

@kekscom
Copy link
Author

kekscom commented Jul 8, 2019

It was causes by excessive use of console.log()
See nodejs/node#3171

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

1 participant