PyArrow Parquet column partitioning

[wanx4910]

I am writing some data to parquet format that requires partitioning the dataset by the column value and the column value contains non-alphanumeric characters. For example, our date format is '2021/08/30' and when the data gets written into the filesystem, the folder structure splits on the '/' character and it looks somehing like 'date=2021'/08/30/somedata.parquet. I was wonder how to get around the folder splitting behaviour, so that the folder would be something like this /'date=2021/08/30'/somedata.parquet

This is the most similar issue report that I can find on the java arrow/parquet repo.
apache/parquet-java#361

Thanks for all the help!

[jorisvandenbossche]

@wanx4910 which version of Arrow are you using? I think this should be fixed in 5.0.0 (https://issues.apache.org/jira/browse/ARROW-12644)

[wanx4910]

@wanx4910 which version of Arrow are you using? I think this should be fixed in 5.0.0 (https://issues.apache.org/jira/browse/ARROW-12644)

PyArrow 0.14.1, but I don't have issues with '-', just '/'. Anytime a '/' is present in the string I want to use to partition a datasets, it does the folder splitting behaviour.

I'll try upgrading the pyarrow version and see if that solves the issue

[westonpace]

I think ARROW-12644 fixes something different. My gut reaction would be to not do this. It seems reasonable to expect that partition columns only contain filesystem-safe paths.

Spark URL encodes non-safe characters (I'm not sure if it does this in all cases or just when using timestamps as a partition column) and ARROW-12644 was making sure we could read these but, as discussed in the JIRA, it isn't clear that we should support writing such paths.

/'date=2021/08/30'/somedata.parquet is not going to be a safe path on all filesystems so I don't think that is a viable alternative. If we were to URL encode paths and you would get 2021%2F08%2F30 which is an odd thing to have in the filesystem but it should at least work. Perhaps we need a URL encoding kernel and then you could partition on that column projected with URL encoding (although I don't think projection support is quite there yet).

Today, as a possible workaround, you could use pyarrow compute to do a replace on / with the character of your choosing: https://arrow.apache.org/docs/cpp/compute.html#string-transforms

[jorisvandenbossche]

Ah, ARROW-12644 indeed only implemented the decoding when reading, not the equivalent encoding when writing. But so if we can read such datasets, we should probably also enable to write them? (will open a JIRA about that)

@wanx4910 To show that we can read values with encoded / (illustrating what @westonpace mentioned above), I created a small dataset with two directories with URL encoded values (using a european date format of 2012/01/01):

In [44]: !ls test_decoding.parquet/
2012%2F01%2F01	2012%2F01%2F02

In [45]: dataset = ds.dataset("test_decoding.parquet/", partitioning=["date"], format="parquet")

In [46]: dataset
Out[46]: <pyarrow._dataset.FileSystemDataset at 0x7f110c345770>

In [47]: dataset.to_table().to_pandas()
Out[47]: 
   b        date
0  1  2012/01/01
1  2  2012/01/02

So when reading, we can properly decode such values.

[jorisvandenbossche]

I opened https://issues.apache.org/jira/browse/ARROW-13813, so we can continue the discussion there.

@wanx4910 although it thus won't yet fix your issue, I would recommend to still upgrade your pyarrow version, since 0.14 is quite old.