Can't read in 1GB data

[OneWind]

I am trying to repeat the code in:
http://wesmckinney.com/blog/?p=635
But I can't read in the data. I have more than 4GB memory available (with 8GB memory in total), but still has the following problem:

In [2]: data = pd.read_csv("P00000001-ALL.csv", index_col=False)
Python(9081,0xacb01a28) malloc: *** mmap(size=24051712) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug

[wesm]

can you give me a backtrace with gdb?

gdb --args python -c "import pandas as pd; data = pd.read_csv('P00000001-ALL.csv', index_col=False)"

then enter r and let it segfault then bt

[OneWind]

Thanks for helping me. This is what I got:

This GDB was configured as "x86_64-apple-darwin"...Reading symbols for shared libraries .
warning: Could not find object file "/Users/builder/work/Python-2.7.3/libpython2.7.a(getbuildinfo.o)" - no debug information available for "./Modules/getbuildinfo.c".

warning: Could not find object file "/Users/builder/work/Python-2.7.3/libpython2.7.a(acceler.o)" - no debug information available for "Parser/acceler.c".

(I got a lot of warnings like these)
...

.. done

(gdb) r
Starting program: /Library/Frameworks/Python.framework/Versions/7.3/bin/python -c import\ pandas\ as\ pd;\ data\ =\ pd.read_csv(P00000001-ALL.csv,\ index_col=False)
Reading symbols for shared libraries ++......................... done

Program received signal SIGTRAP, Trace/breakpoint trap.
0x8fe01030 in __dyld__dyld_start ()
(gdb) bt
#0 0x8fe01030 in __dyld__dyld_start ()
#1 0x00001000 in ?? ()

[wesm]

Can you please try a (free) packaged python distro like EPDFree or Anaconda CE to see if it's a problem with your build of Python? You can pass engine='python' for now as a workaround to still be able to load the file.

[wesm]

Did you have a chance to look any further into this?

[ghost]

carrier lost.

[martinbel]

I have a similar issue with a 1.67 GB file. I've tried engine='python' but it eats all the RAM (16GB). I've left it 10 minutes and it didn't finish.

[Yevgnen]

Any update on this?

[jreback]

this is a long closed issue
if you have a reproducible issue with current pandas open a new issue

[Yevgnen]

I load a 1.4G json and parse it into list of dicts and create a large DataFrame, I can save to the csv by to_csv but I cannot load it by pd.read_csv. Exactly encountered the same issue as OP.

[jreback]

@Yevgnen

reproducible by others, with full code, and versions.

[seralouk]

Hello. I have an 1.2 GB csv (created using MatLab) and now I want to load it using pandas. If I do not specify engine='python' it just freezes. If I set it, it works. Why is that?