Over 256Mb datafile throw err upon opening

[ornic]

Windows x64 Node v5.3.0 and v5.7.0 x64

Datafile contains large portions of data (about 1 Mb for each JSON) updated continuosly. Clean datafile (no deleted entries) has size about 3 Mb.

It is looks like somebody (NEDB?) try to load all file into one string and bumps into nodejs/node#3175 256Mb limit.

May be it will help to set some size limit to force compactification?

Node window (VS 2015 community)

...\src\node_modules\nedb\lib\datastore.js:77
    if (err) { throw err; }
               ^

Error: toString failed
    at Buffer.toString (buffer.js:397:11)
    at tryToString (fs.js:410:15)
    at FSReqWrap.readFileAfterClose [as oncomplete] (fs.js:401:12)

Interactive node

> var Datastore = require('nedb'),
.     db = new Datastore({ filename: './DB/DBSMdata.nedb', timestampData: true, autoload: false });
undefined
> db.persistence.compactDatafile()
undefined
> db.loadDatabase(function (err) {    // Callback is optional
.     console.log(err)
. });
undefined
[Error: toString failed]

[lukasbestle]

See #363. NeDB is not meant to be used with that large databases. You should probably use MongoDB.

[ornic]

Thanks, @lukasbestle, I searched with wrong words.

The DB is not that large (3 Mb of actual data), but I messed up auto compactification and got stuck with huge file I had to edit in UltraEdit32 to restore NEDB ability to work. :) Now that I know, I will add some more checks for size.

But mention something about 256Mb limit of DB filesize in https://github.com/louischatriot/nedb#persistence may be useful for future dummies like me.

[lukasbestle]

Ah, alright. If auto-compaction works for you now, that should be solved. The note in the README is a good idea however. @louischatriot?

[louischatriot]

Hmm adding a check on file size could be an idea but there are a lot of questions here. I'm pretty sure that there is no 256 MB limite on nedb datafiles though, that must be something to do with your system.

[ornic]

As I can see (but I am pretty much newbie in Node) there is a problem in Buffer.js during DB loading and trying to convert all file from Buffer to String in one go. Since one String can't be longer than kMaxLength in V8.

Checking in node.exe (5.7.0)

var len = 268435441;
var blob = new Buffer(len).toString('utf8');

result is (also tried ascii encoding just in case, NEDB files is utf-8)

Error: toString failed
    at Buffer.toString (buffer.js:397:11)
    at repl:1:24
    at REPLServer.defaultEval (repl.js:262:27)
    at bound (domain.js:287:14)
    at REPLServer.runBound [as eval] (domain.js:300:12)
    at REPLServer.<anonymous> (repl.js:427:12)
    at emitOne (events.js:95:20)
    at REPLServer.emit (events.js:182:7)
    at REPLServer.Interface._onLine (readline.js:211:10)
    at REPLServer.Interface._line (readline.js:550:8)

Just for fun tried JSON.stringify(new Buffer(len)); and it threw an exception

FATAL ERROR: invalid array length Allocation failed - process out of memory

As I understood from googling, there is no way to open over256Mb JSON file in node in one go: only streaming by line and parsing into JSON.

[louischatriot]

Final comment on this: if this is indeed an issue that's reproducible, please create a test case according to the bug reporting guidelines, which I will then review!

[ornic]

Sorry, I am new to these guidlines, and if I'm wrong please point me to the proper doc.

I tested in Express v3.4.4, Node 5.7.0. Folder for DB file is ./DB

app.js

var express = require('express');
var http = require('http');
var app = express();

app.set('port', process.env.PORT || 3000);
app.set('views', './views');
app.use(express.logger('dev'));
app.use(express.json());
app.use(express.urlencoded());
app.use(express.methodOverride());
app.use(express.static('./public'));

var stats = require('./routes/index.js');
app.get('/testDB', stats.testDB);

http.createServer(app).listen(app.get('port'), function () {
    // console.dir(process.versions);
    console.log('Express server in mode ' + app.get('env') + ' listening on port ' + app.get('port'));
});

index.js.

function checkDBsize() {
    var fs = require('fs');

    fstat = fs.statSync('./DB/DBtest.nedb');
    var filesize = fstat["size"];

    console.log("DB size is", filesize, "bytes");

    return filesize;
}

function increaseDBsize(db, limit, res) {

    var data = [];
    for (var i = 0; i < 15; i++)
        data.push({index: i, dataSegment: JSON.stringify(data)});

    db.insert({ test: true, dataChunk: data }, function (err, newDoc) {
        if (newDoc) {
            db.remove({ test: true }, { multi: true }, function (err, recNum) {
                if (err) {
                    console.log("error deleting", JSON.stringify(err));
                } else {
                    filesize = checkDBsize();
                    if (filesize >= limit)
                        res.send(filesize + " bytes");
                    else
                        return increaseDBsize(db, limit, res);
                }
            });
        } else {
            console.log("error adding", JSON.stringify(err));
        }
    });
}

exports.testDB = function (req, res) {

    var text2send = "";
    var filesize = 0;
    var DatastoreTest = require('nedb'),
        db = new DatastoreTest({ filename: './DB/DBtest.nedb', timestampData: true, autoload: true });

    filesize = increaseDBsize(db, 350 * 1000000, res); // With 250 all is OK.

}

First run of http://localhost:1337 (I'm using VS2015 community) produces NEDB database file ~335Mb.

console output is:

DB size is 43849712 bytes
DB size is 87699424 bytes
DB size is 131549136 bytes
DB size is 175398848 bytes
DB size is 219248560 bytes
DB size is 263098272 bytes
DB size is 306947984 bytes
DB size is 350797696 bytes
GET /testDB 200 9252ms - 15b

Second run crashes Node.

C:\Users\Roman\Documents\Git\dbsm\src\node_modules\nedb\lib\datastore.js:77
    if (err) { throw err; }
               ^

Error: toString failed
    at Buffer.toString (buffer.js:397:11)
    at tryToString (fs.js:410:15)
    at FSReqWrap.readFileAfterClose [as oncomplete] (fs.js:401:12)
 Press any key to continue...

[louischatriot]

Sorry but this still doesn't adhere to the bug reporting guidelines, see bottom of the readme. Specifically, please remove all unneeded dependencies such as Express to remove external factors, and you sohuld be able to make it so the crash/bad assertion occurs in only one run.

[PetrChalov]

When you try to open NeDb size of 500MB, an exception is thrown.

node_modules/nedb/lib/datastore.js:77
     if (err) {throw err; }
                ^

Error: toString failed
     at Buffer.toString (buffer.js: 397: 11)
     at tryToString (fs.js: 410: 15)
     at FSReqWrap.readFileAfterClose [as oncomplete] (fs.js: 401: 12)

Probably due to the fact that the file size exceeds the maximum Buffer if you are going to deal with this bug?

Thank you.

[markgeraty]

Here is a gist of @ornic's code, simplified down. You can run it as node bugreport.js or just npm test. It will fail with the error on the second run-through.

Are there any plans to accommodate larger file sizes? Or should people be using https://github.com/Ivshti/linvodb3 or mongo instead?

If not, I do agree with some of the previous posters that this should be addressed clearly near the top of the README.

[anthonyettinger]

I'm also seeing this error on db size of 256MB. Looks like no work around?

[jampy]

Seems that the reason is the max string size supported by Chrome, which is 512MB/2: http://stackoverflow.com/q/34957890

[ralyodio]

Can we remove the limit if you're using nedb from node.js on the server? There is no need for a 256MB database limit.

[lukasbestle]

This is not a limit that has been hardcoded into NeDB. It's a limitation of how many characters can be loaded into the string in Node.

NeDB could load the DB by streaming from the file, but this wouldn't solve the issue that large DBs won't perform well. Consider using a "real" DB like MongoDB.