Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CSV.write crashes with large dataframe #735

Closed
hdavid16 opened this issue Aug 29, 2020 · 2 comments
Closed

CSV.write crashes with large dataframe #735

hdavid16 opened this issue Aug 29, 2020 · 2 comments

Comments

@hdavid16
Copy link

See the following examples using CSV 0.6.2. Output file test.csv is empty.

Example #1

julia> using CSV, DataFrames

julia> df=DataFrame(Dict(i=>rand(15) for i in 1:250000));

julia> try
           CSV.write("test.csv",df)
       catch e
           println(e)
       end
UndefVarError(Symbol("53393"))

Example #2

julia> df=DataFrame(Dict(i=>rand(10) for i in 1:300000));

julia> try
           CSV.write("test.csv",df)
       catch e
           println(e)
       end

Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x668dd90c -- jl_uncompress_ast at /cygdrive/d/buildbot/worker/package_win64/build/src\dump.c:2658
in expression starting at REPL[7]:1
jl_uncompress_ast at /cygdrive/d/buildbot/worker/package_win64/build/src\dump.c:2658
jl_compile_linfo at /cygdrive/d/buildbot/worker/package_win64/build/src\codegen.cpp:1192
emit_invoke at /cygdrive/d/buildbot/worker/package_win64/build/src\codegen.cpp:3323
emit_expr at /cygdrive/d/buildbot/worker/package_win64/build/src\codegen.cpp:4139
emit_ssaval_assign at /cygdrive/d/buildbot/worker/package_win64/build/src\codegen.cpp:3851
emit_stmtpos at /cygdrive/d/buildbot/worker/package_win64/build/src\codegen.cpp:4044 [inlined]
emit_function at /cygdrive/d/buildbot/worker/package_win64/build/src\codegen.cpp:6671
jl_compile_linfo at /cygdrive/d/buildbot/worker/package_win64/build/src\codegen.cpp:1257
emit_invoke at /cygdrive/d/buildbot/worker/package_win64/build/src\codegen.cpp:3323
emit_expr at /cygdrive/d/buildbot/worker/package_win64/build/src\codegen.cpp:4139
emit_ssaval_assign at /cygdrive/d/buildbot/worker/package_win64/build/src\codegen.cpp:3851
emit_stmtpos at /cygdrive/d/buildbot/worker/package_win64/build/src\codegen.cpp:4044 [inlined]
emit_function at /cygdrive/d/buildbot/worker/package_win64/build/src\codegen.cpp:6671
jl_compile_linfo at /cygdrive/d/buildbot/worker/package_win64/build/src\codegen.cpp:1257
jl_compile_method_internal at /cygdrive/d/buildbot/worker/package_win64/build/src\gf.c:1889
_jl_invoke at /cygdrive/d/buildbot/worker/package_win64/build/src\gf.c:2153 [inlined]
jl_apply_generic at /cygdrive/d/buildbot/worker/package_win64/build/src\gf.c:2322
open at .\io.jl:296
with at C:\Users\HD\.julia\packages\CSV\vyG0T\src\write.jl:240 [inlined]
#write#72 at C:\Users\HD\.julia\packages\CSV\vyG0T\src\write.jl:167 [inlined]
write at C:\Users\HD\.julia\packages\CSV\vyG0T\src\write.jl:162
#write#67 at C:\Users\HD\.julia\packages\CSV\vyG0T\src\write.jl:154
write at C:\Users\HD\.julia\packages\CSV\vyG0T\src\write.jl:147
jl_apply at /cygdrive/d/buildbot/worker/package_win64/build/src\julia.h:1700 [inlined]
do_call at /cygdrive/d/buildbot/worker/package_win64/build/src\interpreter.c:369
eval_value at /cygdrive/d/buildbot/worker/package_win64/build/src\interpreter.c:458
eval_stmt_value at /cygdrive/d/buildbot/worker/package_win64/build/src\interpreter.c:409 [inlined]
eval_body at /cygdrive/d/buildbot/worker/package_win64/build/src\interpreter.c:799
eval_body at /cygdrive/d/buildbot/worker/package_win64/build/src\interpreter.c:744
jl_interpret_toplevel_thunk at /cygdrive/d/buildbot/worker/package_win64/build/src\interpreter.c:911
jl_toplevel_eval_flex at /cygdrive/d/buildbot/worker/package_win64/build/src\toplevel.c:814
jl_toplevel_eval_flex at /cygdrive/d/buildbot/worker/package_win64/build/src\toplevel.c:764
jl_toplevel_eval at /cygdrive/d/buildbot/worker/package_win64/build/src\toplevel.c:823 [inlined]
jl_toplevel_eval_in at /cygdrive/d/buildbot/worker/package_win64/build/src\toplevel.c:843
eval at .\boot.jl:331
eval_user_input at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\REPL\src\REPL.jl:86
run_backend at C:\Users\HD\.julia\packages\Revise\BqeJF\src\Revise.jl:1184
jl_apply at /cygdrive/d/buildbot/worker/package_win64/build/src\julia.h:1700 [inlined]
do_call at /cygdrive/d/buildbot/worker/package_win64/build/src\interpreter.c:369
eval_value at /cygdrive/d/buildbot/worker/package_win64/build/src\interpreter.c:458
eval_stmt_value at /cygdrive/d/buildbot/worker/package_win64/build/src\interpreter.c:409 [inlined]
eval_body at /cygdrive/d/buildbot/worker/package_win64/build/src\interpreter.c:799
jl_interpret_toplevel_thunk at /cygdrive/d/buildbot/worker/package_win64/build/src\interpreter.c:911
jl_toplevel_eval_flex at /cygdrive/d/buildbot/worker/package_win64/build/src\toplevel.c:814
jl_toplevel_eval at /cygdrive/d/buildbot/worker/package_win64/build/src\toplevel.c:823 [inlined]
jl_toplevel_eval_in at /cygdrive/d/buildbot/worker/package_win64/build/src\toplevel.c:843
eval at .\boot.jl:331
eval_user_input at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\REPL\src\REPL.jl:86
macro expansion at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\REPL\src\REPL.jl:118 [inlined]
#26 at .\task.jl:358
jl_apply at /cygdrive/d/buildbot/worker/package_win64/build/src\julia.h:1700 [inlined]
start_task at /cygdrive/d/buildbot/worker/package_win64/build/src\task.c:687
Allocations: 76336770 (Pool: 76326393; Big: 10377); GC: 75
@hdavid16
Copy link
Author

The problem seems to me when you have too many columns. Writing a transposed dataframe for the examples above works well.

@quinnj
Copy link
Member

quinnj commented Oct 29, 2020

Duplicate of #635

@quinnj quinnj marked this as a duplicate of #635 Oct 29, 2020
@quinnj quinnj closed this as completed Oct 29, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants