-
Notifications
You must be signed in to change notification settings - Fork 27
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DeepMergeWith inserter function causes high memory usage #90
Comments
It is expected that the writer will use a fair bit of memory. You don't provide any information on how you are using the writer, but in terms of what you have provided:
|
Thanks. Yes you are right, I am using the |
Looking at the code, I think it would be possible to get rid of the We don't have a single internal use of |
I am trying to convert 500 MB CSV file into MMDB and the code is super straightforward. It reads the CSV file as a stream and calling |
This writer is built for our use case. It constructs the whole tree in memory, which saves IO and CPU time. Memory is relatively cheap. In terms of the memory usage of your particular use case, I might suggest interning the I have left this particular issue open as there are optimizations that can be made to the |
Hi,
I have been using the mmdbwriter package in golang to insert some records into an MMDB file. I observed that the script to insert records into MMDB consumes a lot of memory. I did some memory profiling using
pprof
and I found that a couple of MMDB writer functions were consuming most of the memory. I have attached a screenshot below for reference.These functions consume around 600MB each whereas the size of the final MMDB file was 146MB. Overall, the program consumed around 3.8GB for producing a file of 146MB. I think these functions, specially the Map.Copy(), stores the records in-memory and is not garbage collected since their references are still in use. This profiling is done just before the writer writes the MMDB file on disk.
Here's how I have defined the MMDB writer:
I am using the
DeepMergeWith
inserter to insert the MMDB recordsThe text was updated successfully, but these errors were encountered: