Incorrect and inconsistency handling of NULL values in ScalarValue for ListArray

[viirya]

Describe the bug

• try_from_array produces inconsistent ScalarValue::List for valid and non-valid (NULL) values

  For NULL value in ListArray, try_from_array produces ScalarValue::List(new_null_array(&DataType::Null, 0)) which isn't consistent with the case when invoking try_from_array on valid value of ListArray.

  Fro valid value, try_from_array produces a ListArray containing single element of the specified value in provided ListArray. For example, if input array is ListArray[[1, 2, 3], [4, 5, 6]]], try_from_array(arr, 0) will produce ListArray[[1, 2, 3]].

  For NULL value, try_from_array currently produces a null array ofDataType::Null type. Not only it is not a ListArray, the data type is not the element type of provided ListArray too.

• ScalarValue::List has same value and datatype for ListArray and nested ListArray's NULL values
  NULL values from ListArray and nested ListArray are the same ScalarValue::List(new_null_array(&DataType::Null, 0)). So their datatypes are the same. This is also incorrect.

To Reproduce

No response

Expected behavior

No response

Additional context

No response

[alamb]

Thank you @viirya -- the way ARRAY datatypes handle null now is non ideal -- they use DataType::Null inconsistenty rather than DataType::List with an appropriate Null literal -- perhaps this is related to #7142

cc @jayzhan211 -- maybe #7142 would be a good simplification to try and work on in the context of moving DataFusion array support forward

[jayzhan211]

cc @jayzhan211 -- maybe #7142 would be a good simplification to try and work on in the context of moving DataFusion array support forward

Great, I was thinking of it too.

[viirya]

No, this is another issue, I think.

This issue is due to incorrect and inconsistency handling of NULL value on ScalarValue::List. It doesn't involve array functions. I added some description to this issue. Please also see the proposed PR.