[Python][Parquet] pa.schema() silently drops metadata if a schema object is passed

[igozali]

Describe the bug, including details regarding any error messages, version, and platform.

Seems like a regression from arrow 13 -> 14

Here's a simple repro script

import pyarrow as pa
import pyarrow.parquet as pq

print(pa.__version__)

metadata = {b"foo": b"bar"}
schema = pa.schema([pa.field("foo", pa.int32())])
wrapped_schema = pa.schema(schema, metadata=metadata)

w = pq.ParquetWriter(
    "foo.parquet", 
    wrapped_schema,
    flavor="spark", 
    compression="snappy"
)
w.close()

s = pq.read_schema("foo.parquet")
print(f"{s.metadata=} {metadata=}")
print(f"{s.metadata == metadata=}")

Running above script on pyarrow 13 and 14 gives these outputs:

(arrow13)
[11:43:42 | last: 43s] (  0) | ~
igozali@host $ python test.py
13.0.0
s.metadata={b'foo': b'bar'} metadata={b'foo': b'bar'}
s.metadata == metadata=True

(arrow13)
[11:43:44 | last: 0s] (  0) | ~
igozali@host $ conda activate arrow14

(arrow14)
[11:43:49 | last: 0s] (  0) | ~
igozali@host $ python test.py
14.0.0
s.metadata=None metadata={b'foo': b'bar'}
s.metadata == metadata=False

Few workarounds:

1. Use schema.with_metadata() instead of pa.schema()
2. Instead of pa.schema(old_schema, metadata=...), passing a list of fields seems to fix it too e.g. pa.schema(list(old_schema), metadata=...)

Component(s)

Parquet, Python

[JacobHayes]

I came across this too - it also appears to also happen if you pass a pa.struct into pa.schema, eg:

$ python3
Python 3.12.2 (main, May  3 2024, 17:50:09) [Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyarrow as pa
>>> pa.__version__
'16.0.0'
>>> pa.schema([pa.field("test", pa.int64())], metadata={"a": "b"})
test: int64
-- schema metadata --
a: 'b'
>>> pa.schema(pa.struct([pa.field("test", pa.int64())]), metadata={"a": "b"})
test: int64

[JacobHayes]

I think it was introduced in #37797, which short-circuits the creation without setting the metadata:

arrow/python/pyarrow/types.pxi

Lines 5093 to 5094 in d68f8e2

	elif hasattr(fields, "__arrow_c_schema__"):
	return Schema._import_from_c_capsule(fields.__arrow_c_schema__())

[jorisvandenbossche]

Issue resolved by pull request 41538
#41538