Change ScalarValue::List to store ArrayRef

[tustvold]

Is your feature request related to a problem or challenge?

Currently ScalarValue::List stores a Vec<Option<ScalarValue>>. This is incredibly inefficient as it stores nullability and type information per nested value. It also makes the type very cumbersome to use, as the data cannot easily be fed into the normal array kernels.

Describe the solution you'd like

I would like to change the type of ScalarValue::List to store ArrayRef

Describe alternatives you've considered

No response

Additional context

See #7353

[Dandandan]

#996 seems to be a similar suggestion

[Dandandan]

Also see discussions here:

#6837 (comment)
#6837 (comment)

[tustvold]

#996 seems to be a similar suggestion

Happy to close this in favor if you prefer, mainly just hoping to bring more attention to this. I know there is a decent amount of ongoing work relating to lists at the moment

[alamb]

I think Change ScalarValue::List to store ArrayRef is a great idea

An ArrayRef is a pretty high overhead way of representing other scalar types but for lists it is different

I also think doing this would allow us to work on / improve the ergonomics of using ListArray in general and set some patterns (like using take and manipulating the list indexes themselves) as well

So 👍 from me

[jayzhan211]

Is anyone working on this? If not, I would like to help with this.

[Dandandan]

Is anyone working on this? If not, I would like to help with this.

I don't think so. I see @tustvold already assigned you on this.

[jayzhan211]

Is it ok to wrap any ArrayRef at the first step?

Not only ListArray(PrimitiveArray(1)) but also PrimitiveArray(1) are wrapped in ScalarValue::List.

In the current implementation.
ScalarValue::Int64(1) can transform to PrimitiveArray(1) and it is reversible.

In my current design,
After ScalarValue::Int64(1) transform to PrimitiveArray(1), it will becomes ScalarValue::List and cant transform back to ScalarValue::Int64(1) via try_from_array().

I think it is fine since we do not need to guarantee the ScalarValue and Array mapping is one-to-one.

[tustvold]

we do not need to guarantee the ScalarValue and Array mapping is one-to-one

I'm not sure I follow what you're suggesting but this seems like a fairly fundamental property