parquet.hadoop.ParquetOutputCommitter.commitJob() throws parquet.io.ParquetEncodingException

[asfimport]

I'm running an example combining Avro, Spark and Parquet (https://github.com/massie/spark-parquet-example), and in the process of updating the library versions, am getting the warning below.

The version of Parquet-Hadoop in the original example is 1.0.0. I am using 1.6.0rc3

The ParquetFileWriter.mergeFooters(Path, List

) method is performing a check to ensure the footers are all for files in the output directory. The output directory is supplied by ParquetFileWriter.writeMetadataFile; in 1.0.0, the output path was converted to a fully qualified output path before the call to mergeFooters, but in 1.6.0rc[2,3] that conversion happens after the call to mergeFooters. Because of this, the check within merge footers is failing (the URI for the footers starts with file:, but not the URI for the root path does not)

Here is the warning message and stacktrace.

Oct 30, 2014 9:11:31 PM WARNING: parquet.hadoop.ParquetOutputCommitter: could not write summary file for /tmp/1414728690018-0/output
parquet.io.ParquetEncodingException: file:/tmp/1414728690018-0/output/part-r-00000.parquet invalid: all the files must be contained in the root /tmp/1414728690018-0/output
at parquet.hadoop.ParquetFileWriter.mergeFooters(ParquetFileWriter.java:422)
at parquet.hadoop.ParquetFileWriter.writeMetadataFile(ParquetFileWriter.java:398)
at parquet.hadoop.ParquetOutputCommitter.commitJob(ParquetOutputCommitter.java:50)
at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:936)
at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(PairRDDFunctions.scala:832)
at com.zenfractal.SparkParquetExample$.main(SparkParquetExample.scala:72)
at com.zenfractal.SparkParquetExample.main(SparkParquetExample.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)

Reporter: Chris Albright

Original Issue Attachments:

• PARQUET-124-test

_{Note: This issue was originally created as PARQUET-124. Please see the migration documentation for further details.}

[asfimport]

Ryan Blue / @rdblue:
Looks like the problem is that we're matching a URI against a path. The check should be URI.getPath().startWith(dirPath). I'll submit a PR today unless you want to take care of it, Chris. The fix should be on line 422 of ParquetFileWriter.

[asfimport]

Chris Albright:
I was trying to create a breaking test for it last night, but to no avail. Probably need to get a bit more experience with the codebase. I went ahead and submitted the PR, but without a test I don't feel great about it.

[asfimport]

Ryan Blue / @rdblue:
Great, now it'll look bad if I don't include a test. :)

[asfimport]

Ryan Blue / @rdblue:
Chris, I'm attaching a test that produces the same error that you've posted. I ended up making the mergeFooters method accessible within the package to add it. Could you check your PR and push a new version with this test included so we can see it passes tests in Travis?

Also, is there a better way to check that the path than to use String#startWith? If there is a Path or URI method that does it, that would be preferred. That way we don't run into problems trying to merge hdfs:/ and file:/ URIs. That shouldn't ever happen though.

[asfimport]

Chris Albright:
@rdblue, I was not able to find a way compare to the path prefixes without using String#startsWith. It does seem very unlikely that there would ever exist a situation where the footers come from a different filesystem than the root, since the footers are read out of the root to begin with :)

I thought of comparing the scheme as well, but I was not able to make that work since the scheme was missing from the root path URI. If its important, I've got some other ideas, but I'm not sure its within scope of this issue. I'll go ahead and create a new one and work off of that.

Pull request is updated, and passing in Travis.

[asfimport]

Ryan Blue / @rdblue:
Issue resolved by pull request 79
https://github.com/apache/incubator-parquet-mr/pull/79

[asfimport]

Alex Nastetsky:
Still seeing this issue, at least when running locally. I think it needs to do something like FileSystem.get(new URI("file:///"), new Configuration()).resolvePath(X) on both "root" and "new Path(footerPath)", except using the correct file system.

[asfimport]

swetha k:
[~blue@cloudera.com]

I still see the issues. Please see the Warning below: Which version to be used to avoid this? I use 1.6.0 of parquet-avro.

    <dependency>
        <groupId>com.twitter</groupId>
        <artifactId>parquet-avro</artifactId>
        <version>1.6.0</version>
    </dependency>

How is this issue resolved? I still see the following issue when I try to save my parquet file.
Nov 8, 2015 11:35:39 PM WARNING: parquet.hadoop.ParquetOutputCommitter: could not write summary file for active_sessions_current
parquet.io.ParquetEncodingException: maprfs:/user/testId/active_sessions_current/part-r-00142.parquet invalid: all the files must be contained in the root active_sessions_current
at parquet.hadoop.ParquetFileWriter.mergeFooters(ParquetFileWriter.java:422)
at parquet.hadoop.ParquetFileWriter.writeMetadataFile(ParquetFileWriter.java:398)
at parquet.hadoop.ParquetOutputCommitter.commitJob(ParquetOutputCommitter.java:51)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply$mcV$sp(PairRDDFunctions.scala:1056)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:998)

[asfimport]

Ryan Blue / @rdblue:
[~swethakasireddy], it looks like this wasn't completely addressed by the fix above. [~terrasect] had a problem with it as well. Would one of you be willing to open a new issue for the current problem? Then we can work on getting it fixed. Thanks!

[asfimport]

swetha k:
@rdblue

I can create a JIRA issue for this. Just to confirm, this is just a Warning message right and would not affect the functionality in any case?

[asfimport]

swetha k:
@rdblue

Following is the new issues.

https://issues.apache.org/jira/browse/SPARK-11620