Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bad behavior when using some forms of Data.stream! #69

Open
ExpandingMan opened this issue Jan 10, 2018 · 0 comments
Open

bad behavior when using some forms of Data.stream! #69

ExpandingMan opened this issue Jan 10, 2018 · 0 comments

Comments

@ExpandingMan
Copy link
Collaborator

I'm getting a few errors when doing Data.stream!(src, snk) where snk = Feather.Sink("filename.feather", Data.schema(src)) and then calling Data.close!. There does not seem to be a simple fix. It is really nice to be able to use this form, because it is the easiest case for writing API's that can use a variety of different sinks.

See the following MWE

using DataFrames
using Feather

df = DataFrame(A=Union{DateTime,Missing}[DateTime(), DateTime(), missing], B=rand(3), C=rand(Int64,3))

snk = Feather.Sink("test1.feather", Data.schema(df))                                    

Data.stream!(df, snk)

Data.close!(snk)

On the last line, this gives:

ERROR: ArgumentError: reducing over an empty collection is not allowed
Stacktrace:
 [1] mr_empty_iter(::Function, ::Function, ::Base.Generator{Array{Union{DateTime, Missings.Missing},1},Missings.#ismissing}, ::Base.EltypeUnknown) at ./reduce.jl:257
 [2] mapfoldl(::Base.#identity, ::Function, ::Base.Generator{Array{Union{DateTime, Missings.Missing},1},Missings.#ismissing}) at ./reduce.jl:69
 [3] nullcount(::Array{Union{DateTime, Missings.Missing},1}) at /home/expandingman/.julia/v0.6/Feather/src/Feather.jl:329
 [4] close!(::Feather.Sink{DataFrames.DataFrameStream{Tuple{Array{Union{DateTime, Missings.Missing},1},Array{Float64,1},Array{Int64,1}}}}) at /home/expandingman/.julia/v0.6/Feather/src/Feather.jl:465

This seems related to a deeper issue of the snk.df field being an empty DataFramesStream.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant