[Python] pyarrow.Array.from_pandas should take a type parameter

[asfimport]

It's convenient to infer the data type so that users can just write pyarrow.Array.from_pandas(arr), however, sometimes users want fine-grained control, e.g., if we have an object type numpy array, whose values are all null. When we convert it to an arrow column vector, we may need a specific type instead of NullType.

Reporter: Wenchen Fan / @cloud-fan
Assignee: Wes McKinney / @wesm

Related issues:

• [C++] Type casting function kernel suite (is part of)
• [Python] Efficient construction of arrays from non-pandas 1D NumPy arrays (relates to)
• [C++] Implement safe casts for primitive types (depends upon)

_{Note: This issue was originally created as ARROW-1156. Please see the migration documentation for further details.}

[asfimport]

Wes McKinney / @wesm:
Note that this functionality already exists in the {{pyarrow.array} function

In [8]: arr
Out[8]: array([None, None, None, None, None, None, None, None, None, None], dtype=object)

In [9]: pa.array(arr)
Out[9]: 
<pyarrow.lib.NullArray object at 0x7f10bd026548>
[
  NA,
  NA,
  NA,
  NA,
  NA,
  NA,
  NA,
  NA,
  NA,
  NA
]

In [10]: pa.array(arr, type=pa.float64())
Out[10]: 
<pyarrow.lib.DoubleArray object at 0x7f10bd026688>
[
  NA,
  NA,
  NA,
  NA,
  NA,
  NA,
  NA,
  NA,
  NA,
  NA
]

[asfimport]

Wenchen Fan / @cloud-fan:
yea that's true, but its functionality is quite limited, a lot of data types are not supported, e.g.

>>> pa.array([None], type=pa.int32())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pyarrow/array.pxi", line 1051, in pyarrow.lib.array (/Users/travis/build/xhochy/pyarrow-macos-wheels/arrow/python/build/temp.macosx-10.6-intel-2.7/lib.cxx:19136)
  File "pyarrow/error.pxi", line 66, in pyarrow.lib.check_status (/Users/travis/build/xhochy/pyarrow-macos-wheels/arrow/python/build/temp.macosx-10.6-intel-2.7/lib.cxx:7124)
pyarrow.lib.ArrowNotImplementedError: NotImplemented: No type converter implemented for int32

BTW if I already have a numpy array, it's less efficient if I have to convert numpy array to python list and call pyarrow.array

[asfimport]

Wes McKinney / @wesm:
Agreed. We need some help here. Our intent is for pyarrow.array to be the main constructor for array data that did not originate from pandas – e.g. many NumPy arrays should be zero-copy constructable (unless they are object dtype, non-contiguous, etc.). Related: ARROW-838

[asfimport]

Wenchen Fan / @cloud-fan:
BTW, I also tried pa.RecordBatch.from_pandas with a schema parameter, however it fails with ArrowNotImplementedError: NotImplemented: NumPy type casts not yet implemented

[asfimport]

Wenchen Fan / @cloud-fan:
I just realized pyarrow.Array.from_pandas do have a type parameter by looking at source code, guys can you fix the official document https://arrow.apache.org/docs/python/generated/pyarrow.Array.html#pyarrow.Array ? It has style issues, and doesn't mention the type parameter, and has no explanation for the parameters.

[asfimport]

Wes McKinney / @wesm:
Marking for 0.6.0 to make some improvements to this functionality (and documentation)

[asfimport]

Wes McKinney / @wesm:
The generated documentation has been much improved in 0.5.0. This particular issue needs to be expanded into more specific JIRAs, since there are many possible features to implement for type casting and so forth. Moving this off 0.6.0 for now

[asfimport]

Wes McKinney / @wesm:
Issue resolved by pull request 1063
#1063