Avoid leaking Hadoop API to downstream libraries

[asfimport]

Parquet currently leaks the Hadoop API by requiring callers to pass Path and Configuration instances, and by using Hadoop codecs. InputFile and SeekableInputStream add alternatives to Hadoop classes in some parts of the read path, but this needs to be extended to the write path and to avoid passing options through Configuration.

Reporter: Ryan Blue / @rdblue
Assignee: Ryan Blue / @rdblue

Related issues:

• make it easy to read and write parquet files in java without depending on hadoop (is part of)

PRs and other links:

• PR #429

_{Note: This issue was originally created as PARQUET-1142. Please see the migration documentation for further details.}

[asfimport]

Ryan Blue / @rdblue:
Merged #429. Thanks to @zivanfi and @gszadovszky for reviewing!

[asfimport]

Werner Daehn / @wernerdaehn:
Thanks Ryan, I believe this is one of the most important areas where Parquet can get better.

Do you happen to have some sample code on how to use that? My current approach is to extend ParquetWriter including the inner Builder class extending ParquetWriter.Builder<T, Builder>.
Where I am stuck is the Builder.WriteSupport with its Hadoop Configuration parameter. It has to be implemented but you talk about ParquetReadOptions?
And my class needs a constructor but all visible super()'s are Hadoop based.
Any hints would be greatly appreciated.

Background:

In my world there is no such thing as Hadoop but many different kinds of file storage options, HDFS being one of those. Therefore I would be in favor of removing all Hadoop related stuff from Parquet and rather have a java.io.OutputStream (=org.apache.parquet.io.OutputFile) which then can be used to store files in different locations, HDFS full client, HDFS web client, S3, local file system.

My ultimate goal for the moment is to have a AvroParquetWriter without Hadoop to save to a local file system in a streaming way. This would then be used to consume the Kafka data of the last 24 hours and write one large Parquet file.
Next step would be a matching reader that does pushdown simple filter criteria. Best would be one that allows even primary key lookups in a reasonable amount of time.

[asfimport]

Hadrien Kohl:
 

Hi. I've faced the same problems and I still need to drag in hadoop dependencies to be able to use ParquetWriter and ParquetReader. 

I am wrapping SeekableByteChannel into InputFile/OutputFile so that I can call

 

ParquetWriter.builder(new Wrapper(Files.newByteChannel("/foo/bar.parquet")));

 

Do you plan to work on this further? I'd be happy to help.

I saw prestodb also implemented another reader/writer.

[asfimport]

Ryan Blue / @rdblue:
The next steps for this are to get compression working without relying on Hadoop. After that, it is a matter of some fairly simple refactoring of the file writer. But that refactoring doesn't help much unless the compression implementations also don't depend on Hadoop.

[asfimport]

Hadrien Kohl:
Right, I think this project has dropin replacements for hadoop CompressionCodecs. It could be a good starting point.
https://github.com/airlift/aircompressor

Is anyone working on this? I could try to get into it if that is part of the roadmap? @rdblue do you know others that would like to get involved?