Incompatibility with hdfs.open

[ephes]

Hi,

I tried to read a parquet file directly from hdfs using pyarrow, but if I set the open_with to hdfs.open, it doesn't seem to work:

from fastparquet import ParquetFile
pf = ParquetFile(hdfs_path, open_with=hdfs.open)

---------------------------------------------------------------------------
ArrowIOError                              Traceback (most recent call last)
~/miniconda3/envs/ro/lib/python3.5/site-packages/fastparquet/api.py in __init__(self, fn, verify, open_with, sep, root)
     92                 self.fn = fn2
---> 93                 with open_with(fn2, 'rb') as f:
     94                     self._parse_header(f, verify)

io.pxi in pyarrow.lib._HdfsClient.open()

error.pxi in pyarrow.lib.check_status()

ArrowIOError: IOError: Unable to open file /user/jwersdoerfer/amoma/queries/ymd=20170126/locale=CA/000000_0/_metadata

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
<ipython-input-26-c56d7ca809d4> in <module>()
      1 from fastparquet import ParquetFile
----> 2 pf = ParquetFile(hdfs_path, open_with=hdfs.open)

~/miniconda3/envs/ro/lib/python3.5/site-packages/fastparquet/api.py in __init__(self, fn, verify, open_with, sep, root)
     97                 self.fn = fn
     98                 with open_with(fn, 'rb') as f:
---> 99                     self._parse_header(f, verify)
    100         if not self.row_groups:
    101             self.file_scheme = 'empty'

~/miniconda3/envs/ro/lib/python3.5/site-packages/fastparquet/api.py in _parse_header(self, f, verify)
    113             if verify:
    114                 assert f.read(4) == b'PAR1'
--> 115             f.seek(-8, 2)
    116             head_size = struct.unpack('<i', f.read(4))[0]
    117             if verify:

TypeError: seek() takes exactly one argument (2 given)

[martindurant]

It seems arrow's seek function is incomplete - I would raise an issue with them. That form or seek says "8 bytes before the end of the file".
You could instead try hdfs3, which is known to work.

[wesm]

Looks like we need to implement the second argument of file.seek in pyarrow.lib.HdfsFile.seek. I opened https://issues.apache.org/jira/browse/ARROW-1287

[ephes]

@martindurant hdfs3 worked and I'll keep it in mind as a fallback method. But I think I'm using pyarrow directly, because it was also 3 times faster on my parquet file. But thanks :).

[martindurant]

Would be interested to see your benchmark - the types of data, and profiling you might do (if you have the time and motivation). If you are happy as things are, please close this issue.

[wesm]

I submitted a patch for the seek issue; I will test it out with fastparquet to make sure all is a-OK