DictEncoded doesn't write as DictEncoded #116

dmbates · 2021-01-29T20:28:06Z

Arrow.toarrowvector of a Arrow.DictEncoded type returns an Arrow.List.

julia> atbl = Arrow.Table(fnm);

julia> typeof(atbl.a)
Arrow.DictEncoded{String, UInt8, Arrow.List{String, Int32, Vector{UInt8}}}

julia> Arrow.toarrowvector(atbl.a)
6-element Arrow.List{String, Int32, Arrow.ToList{UInt8, true, String, Int32}}:
 "a"
 "a"
 "a"
 "b"
 "b"
 "b"

Perhaps it doesn't detect that Arrow.DictEncoded is a factor-like (in the R sense) object.

I can look at this if you wish but I'm not entirely sure where

The text was updated successfully, but these errors were encountered:

…ng issues Fixes #117, #116, and #113. For #116, we just need to special case if user happens to pass in a DictEncoded themselves. We need to pass it through to the `toarrowvector` method that no-ops. For #113, we require the new functionality in PooledArrays that allows passing the `signed` and `compress` keyword arguments to ensure we get signed refs for our dict encoding. For #117, we add CategoricalArrays as a test dependency and ensure that if it contains any `missing` value, we *don't* recode the indices values down by 1, since the `missing` ref is 0, so other refs can already be considered "offsets". If there are no `missing`, then we still need to recode down since refs should always start from 0 in arrow format.

#119) * Rework dict encoding of PooledArray/CategoricalArray to fix outstanding issues Fixes #117, #116, and #113. For #116, we just need to special case if user happens to pass in a DictEncoded themselves. We need to pass it through to the `toarrowvector` method that no-ops. For #113, we require the new functionality in PooledArrays that allows passing the `signed` and `compress` keyword arguments to ensure we get signed refs for our dict encoding. For #117, we add CategoricalArrays as a test dependency and ensure that if it contains any `missing` value, we *don't* recode the indices values down by 1, since the `missing` ref is 0, so other refs can already be considered "offsets". If there are no `missing`, then we still need to recode down since refs should always start from 0 in arrow format. * PooledArrays 1.0 compat * Update src/arraytypes/dictencoding.jl Co-authored-by: Milan Bouchet-Valat <nalimilan@club.fr> * Check refpool * Fix test Co-authored-by: Milan Bouchet-Valat <nalimilan@club.fr>

quinnj · 2021-01-31T06:19:16Z

Fixed in #119

apache#119) * Rework dict encoding of PooledArray/CategoricalArray to fix outstanding issues Fixes apache#117, apache#116, and apache#113. For apache#116, we just need to special case if user happens to pass in a DictEncoded themselves. We need to pass it through to the `toarrowvector` method that no-ops. For apache#113, we require the new functionality in PooledArrays that allows passing the `signed` and `compress` keyword arguments to ensure we get signed refs for our dict encoding. For apache#117, we add CategoricalArrays as a test dependency and ensure that if it contains any `missing` value, we *don't* recode the indices values down by 1, since the `missing` ref is 0, so other refs can already be considered "offsets". If there are no `missing`, then we still need to recode down since refs should always start from 0 in arrow format. * PooledArrays 1.0 compat * Update src/arraytypes/dictencoding.jl Co-authored-by: Milan Bouchet-Valat <nalimilan@club.fr> * Check refpool * Fix test Co-authored-by: Milan Bouchet-Valat <nalimilan@club.fr>

quinnj mentioned this issue Jan 30, 2021

Rework dict encoding of PooledArray/CategoricalArray to fix outstandi… #119

Merged

quinnj closed this as completed Jan 31, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DictEncoded doesn't write as DictEncoded #116

DictEncoded doesn't write as DictEncoded #116

dmbates commented Jan 29, 2021 •

edited

Loading

quinnj commented Jan 31, 2021

DictEncoded doesn't write as DictEncoded #116

DictEncoded doesn't write as DictEncoded #116

Comments

dmbates commented Jan 29, 2021 • edited Loading

quinnj commented Jan 31, 2021

dmbates commented Jan 29, 2021 •

edited

Loading