Round trips through ScalarValue
's sometimes don't preserve types (e.g. change types from DictionaryArray
)
#2874
Labels
bug
Something isn't working
Describe the bug
Various parts of the DataFusion codebase assume that the transformation between
ScalarValue
<-->Array
have the same datatype. This would seem to be a reasonable assumption, however it does not hold for at least forDictionaryArrays
For example, a
ScalarVaule
that is converted to an array,cast
ed to aDictionaryArray<_>
due to coertion rules, and then converted back to aScalarVaule
. When that supposedly castScalarValue
is converted back to an Array, it does not maintain its Dictionary encoding, instead it results in a DataType::Utf8To Reproduce
Running this function results in
Expected behavior
Test case should pass
Additional context
I am not sure if it makes sense to add a
ScalarValue::Dictionary
type variant, or perhaps add ais_dictionary
flag or something else, or maybe even just not assume aScalarValue
can be round tripped and maintain its data typeThis is the root cause of #2873 -- I added a patch for that particular case but this problem can occur elsewhere
The text was updated successfully, but these errors were encountered: