Most efficient way to add/update multiple entries

[Damien455]

Hi ALl, currently using TinyRecord and removing any entires from the database that match certain fields (3 field match) of the new data, then added all the new data on mass by using the add multiple entries by using tr.insert_multiple(documents).

Is this the quickest way or is there a better way, as it still taking a while to write to the database? Much longer than if I was just writing JSON to a text file.

Many Thanks

D

[Damien455]

also looks like my use of tr.remove if not working

using the format tr.remove((querry.name == x )& (querry.id == y)

they use the format tr.remove(where('invalid') == True) but I could not seem to get where to work when using multiple fields

cheers

[msiemens]

Well, if you're writing JSON to a text file, you seem to already have finished all data transformations and manipulations, in which case i wouldn't surprise me if it's faster than TinyDB 🙂 Now the thing with TinyDB is that it writes data to disk after every create/update/delete operation. If you run three delte operations one after the other, TinyDB will write all data to disk after each operation. While this has a performance cost, it helps minimizing the impact of data loss e.g. if the program crashes in the middle of an operation.

But: You can trade off safety for performance e.g. by using the CachingMiddleware which only writes to disk after a configurable number of data manipulation operations (or manually). Maybe this would help you?