Error reading some files after PARQUET-77 bytebuffer read path

[asfimport]

This issue is based on a discussion on the list started by @danielcweeks

Full discussion:
https://mail-archives.apache.org/mod_mbox/parquet-dev/201512.mbox/%3CCAMpYv7C_szTheua9N95bXvbd2ROmV63BFiJTK-K-aDNK6ZNBKA%40mail.gmail.com%3E

From the thread (he later provided a small repro file that is attached here):

Just wanted to see if you or anyone else has run into problems reading
files after the ByteBuffer patch. I've been running into issues and have
narrowed it down to the ByteBuffer commit using a small repro file (written
with 1.6.0, unfortunately can't share the data).

It doesn't happen for every file, but those that fail give this error:

can not read class org.apache.parquet.format.PageHeader: Required field
'uncompressed_page_size' was not found in serialized data! Struct:
PageHeader(type:null, uncompressed_page_size:0, compressed_page_size:0)

Reporter: Jason Altekruse / @jaltekruse
Assignee: Jason Altekruse / @jaltekruse

Related issues:

• Release Parquet-mr 1.9.0 (blocks)

Original Issue Attachments:

• bytebyffer_read_fail.gz.parquet

PRs and other links:

• PR
• PR #349

_{Note: This issue was originally created as PARQUET-400. Please see the migration documentation for further details.}

[asfimport]

Jason Altekruse / @jaltekruse:
@danielcweeks I built parquet at the bytebuffer commit (6b605a4) and tried reading this file using parquet-tools as well as Drill. Neither gave me the exception reading this file, are you reading the file off of local disk? We have seen some different behaviors reading files from different FileSystem implementations, as some of them choose to take advantage of the ability to return a subset of a requested read length. In a few cases in the Drill parquet reader we had neglected to handle this case and it showed up when we started querying files on S3. That being said, I don't even know if you can provide a remote path the the parquet-tools, so I assume you were just reading this off of local disk, in which case we'll have to do some more digging.

Let me know if you can think of any config that may be different on your machine or any other leads.

[asfimport]

Daniel C. Weeks / @danielcweeks:
@jaltekruse Thanks for creating the JIRA.

I tested the attached file with Hadoop 2.4 (Ubuntu 14.04.3 LTS) against S3 and HDFS and both failed. I just tested it against the local filesystem and it was able to read it.

The larger example I have can't be read with S3, HDFS, or local. I'll try to create an example of this and test it across different versions of hadoop.

[asfimport]

Jason Altekruse / @jaltekruse:
I'll throw the file into an S3 bucket and read it from there, should repro the issue. Don't worry too much about creating a second repro on the local fs right away. I'll update you if I get stuck.

[asfimport]

Jason Altekruse / @jaltekruse:
@danielcweeks @zhenxiao apologies for the delay on this, I finally got a chance to try to debug it further and I unfortunately am receiving a different error when reading the file from HDFS.

I did not see any issue reading the file from a 2.6.0 cdh cluster through parquet-cat. I created a test that used the example GroupReadSupport and I saw the following error [1]. I'm confused though why this wouldn't have been reported by parquet-tools as well.

I had some trouble trying to read the file from s3, can you give me a little more info about your setup, are you using jets3t? What version are you using to read the file? Are you using s3:// s3a:// or s3n:// ?

[1]
org.apache.parquet.io.ParquetDecodingException: Can not read value at 0 in block -1 in file hdfs://h011.d.drem.io/tmp/bytebyffer_read_fail.gz.parquet
at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:243)
at org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:125)
at org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:129)
at org.apache.drill.TestExampleQueries.random(TestExampleQueries.java:49)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Caused by: org.apache.parquet.io.ParquetDecodingException: more than one dictionary page in column [value, bag, array] DOUBLE
at org.apache.parquet.hadoop.ParquetFileReader$Chunk.readAllPages(ParquetFileReader.java:597)
at org.apache.parquet.hadoop.ParquetFileReader.readNextRowGroup(ParquetFileReader.java:541)
at org.apache.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:130)
at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:214)

[asfimport]

Daniel C. Weeks / @danielcweeks:
@jaltekruse We use EMR for our hadoop distro currently (AMI 3.9 / Hadoop 2.4). We're reading the file with the EmrFileSystem from S3, but see the same error reading from hdfs.

You could always configure S3AFileSystem if you're using 2.6 (2.7.1 would be better though). I'm also surprised that parquet-cat didn't show the problem.

[asfimport]

Jason Altekruse / @jaltekruse:
@danielcweeks can you please try out the branch on the PR? I believe I figured out both the original issue as well as the secondary issue that was coming up when I ran with a different hadoop version. Both are explained by the reads to the FSDataInputSteam API not returning all requested data. Before the bytebuffer changes the readFully() method was used to avoid this problem, there is no equivalent in the 2.x API for reading into a ByteBuffer and we had missed filling in this functional gap ourselves.

The second error message was related to a different compatibility codepath for some old semi-corrupt files that could be falsely triggered by this incomplete read, this would only happen before the error you had reported if a certain subset of the column chunk had been returned. If enough was returned it could avoid this condition and make it right to the error you reported, explaining the inconsistent behavior.

#306

[asfimport]

Daniel Weeks / @danielcweeks:
@jaltekruse I've made a pull request to your branch to add an option to blacklist S3 (or any filesystem scheme) that currently doesn't work well with bytebuffer read path.

I've verified this works.

[asfimport]

Daniel Weeks / @danielcweeks:
@jaltekruse If you have a chance, could you take a look at my PR(jaltekruse#1). I'd like to close this one out if possible.

[asfimport]

Jason Altekruse / @jaltekruse:
Was just taking a look again. I'll update my PR against the main repo with your commit shortly.

[asfimport]

Piyush Narang / @piyushnarang:
Thought I'd chime in as we ran into this issue as well when we tried to cut an internal Parquet release with what's currently the latest sha at master. We are running plain HDFS (not on S3) think based off version 2.6.0. Would be great if the fix handles hdfs as well. Looking at the current PR (https://github.com/apache/parquet-mr/pull/306/files) seems like the blacklist is only s3 based.

[asfimport]

Daniel C. Weeks / @danielcweeks:
@piyushnarang We uncovered that this isn't related to S3 and shouldn't be handled through blacklisting.

The underlying issue is related to buffer allocations made in a tight loop and @jaltekruse is looking into the fix.

[asfimport]

Ferdinand Xu / @winningsix:
@danielcweeks @jaltekruse Is there any JIRA tracking the fix for this issue? I can easily reproduce it by executing the following Hive SQL after depending on the upstream code. Or any workaround for it?

CREATE EXTERNAL TABLE customer_temporary
  ( c_customer_sk             bigint              --not null
  , c_customer_id             string              --not null
  , c_current_cdemo_sk        bigint
  , c_current_hdemo_sk        bigint
  , c_current_addr_sk         bigint
  , c_first_shipto_date_sk    bigint
  , c_first_sales_date_sk     bigint
  , c_salutation              string
  , c_first_name              string
  , c_last_name               string
  , c_preferred_cust_flag     string
  , c_birth_day               int
  , c_birth_month             int
  , c_birth_year              int
  , c_birth_country           string
  , c_login                   string
  , c_email_address           string
  , c_last_review_date        string
  )
ROW FORMAT DELIMITED FIELDS TERMINATED BY '|'
STORED AS TEXTFILE LOCATION 'file:///tmp/data/';

Select count(*) from customer_temporary;

DROP TABLE IF EXISTS customer2;

CREATE TABLE customer2
STORED AS PARQUET
AS
SELECT * FROM customer_temporary;
SELECT count(*) from customer2;

[asfimport]

Daniel C. Weeks / @danielcweeks:
@winningsix This is the JIRA tracking the issue.

This issue doesn't affect any currently released version of parquet. Also, using anything prior to the bytebuffer patch should still work.