[SPARK-54568][PYTHON] Avoid unnecessary pandas conversion in create dataframe from ndarray

[zhengruifeng]

What changes were proposed in this pull request?

Avoid unnecessary pandas conversion in create dataframe from ndarray

Why are the changes needed?

before:
ndarray -> pandas dataframe -> arrow data

after:
ndarray -> arrow data

and will be consistent with connect mode:

spark/python/pyspark/sql/connect/session.py

Lines 675 to 706 in 40ba971

	elif isinstance(data, np.ndarray):
	if _cols is None:
	if data.ndim == 1 or data.shape[1] == 1:
	_cols = ["value"]
	else:
	_cols = ["_%s" % i for i in range(1, data.shape[1] + 1)]
	if data.ndim == 1:
	if 1 != len(_cols):
	raise PySparkValueError(
	errorClass="AXIS_LENGTH_MISMATCH",
	messageParameters={
	"expected_length": str(len(_cols)),
	"actual_length": "1",
	},
	)
	_table = pa.Table.from_arrays([pa.array(data)], _cols)
	else:
	if data.shape[1] != len(_cols):
	raise PySparkValueError(
	errorClass="AXIS_LENGTH_MISMATCH",
	messageParameters={
	"expected_length": str(len(_cols)),
	"actual_length": str(data.shape[1]),
	},
	)
	_table = pa.Table.from_arrays(
	[pa.array(data[::, i]) for i in range(0, data.shape[1])], _cols
	)

Does this PR introduce any user-facing change?

no

How was this patch tested?

ci

Was this patch authored or co-authored using generative AI tooling?

no

[zhengruifeng]

thanks, merged to master