Clarify parquet-format specification for LIST and MAP structures.

[asfimport]

There are incompatibilities in the way that some parquet object models translate nested structures annotated by LIST and MAP / MAP_KEY_VALUE. We need to define clearly what the structures should look like and how to interpret existing structures, including what must be supported to read current parquet-avro, parquet-thrift, etc. files.

Reporter: Ryan Blue / @rdblue
Assignee: Ryan Blue / @rdblue

Related issues:

• Implement nested type read rules in parquet-thrift (is depended upon by)

_{Note: This issue was originally created as PARQUET-113. Please see the migration documentation for further details.}

[asfimport]

Ryan Blue / @rdblue:
Added PR #17 with additions to the spec for LIST and MAP.

[asfimport]

Sergio Peña / @spena:
Old versions of Hive (<= 0.12) write Map types different than the proposed in this document.

optional group m1 (MAP_KEY_VALUE) {
	repeated group map {
		required binary key;
		optional binary key;
	}
}	

The above is an example of the Map schema written by Hive 0.12. When doing an upgrade to Hive 0.14, then Hive fails to read the file with the old schema. We should be backwards compatible to this schema as well.

[asfimport]

Ryan Blue / @rdblue:
Thanks, @spena! It looks like we need to add another backward-compatibility rule to the spec for maps written with Hive. I think all we would need to do is require that MAP_KEY_VALUE can be used in place of the MAP annotation.

[asfimport]

Philippe Girolami:
@rdblue We've been hit with these issues using parquet-proto. We'd like to know whether changing our "raw" protobuf data model to fit the 2 or 3 level structure for arrays is a good idea : we're afraid once PARQUET-113 gets implemented in parquet-protobuf then we will end up with a 4 or 6 level structure. Any thoughts on this ?

[asfimport]

Ryan Blue / @rdblue:
Thanks for taking a look, [~pgirolami]. The plan with parquet-proto is to not do any translation and to not add MAP or LIST annotations at all. The spec states that unannotated repeated groups should be converted to required lists of required items, so you would end up with the exact same structure you work with in protobuf if you read from parquet-avro or the other object models. If that's something we should change, please let me know. Otherwise, I think that addresses your concern that your structure will be inflated, right?

[asfimport]

Philippe Girolami:
Appreciate the quick reply @rdblue ! It's quite clear.

1. Based on my testing and understanding, working with Lists using these 4 technologies is not currently possible (Protobuf 2.5/Parquet 1.6.0rc3/Hive 0.14/Spark 2.2),

• SparkSQL doesn't understand 3-level lists unless they are annotated (LIST) and there is no way to annotate the parquet file built from protobuf
• so I'm using 2-level lists in the proto schema using the appropriate field naming-scheme (which gives me the additional benefit of allowing me to null the list)
• but Hive doesn't understand 2-level lists so I can't read the table in HIVE

  //2-level encoding, written to Parquet using Spark. Can be read in Spark 2.2 but not in Hive 0.14
  //this is a fake protracted example...
  message Wheel {
  	required	bool	is_front = 1;
  	required	bool is_left	= 2;
  	optional	int32	size = 3;
  	optional	string	color = 4;
  }
  message WheelInner {
  	repeated Wheel array = 1;
  }
  message Car {
  	required uint64 id = 1;
  	required string	 make = 2;
  	optional WheelInner wheels = 3;
  }

I see that you implemented HIVE-8909 : since the specs include handling "old" 2-level data, I'm assuming that once Hive 0.15 is released, I'll be able to read the data. I'm guessing that's what HiveCollectionConverter. isElementType() does.

2. I've been able to use Maps by building the protobuf schema as seen below. I suppose that's why you didn't have to test on special field names for Maps too : "map", "keyvalues".```Java
   //3-level encoding, written to Parquet using Spark. Can be read in Spark 2.2 AND Hive 0.14
   message MapEntryOk {
   required string key = 1;
   optional string value = 2;
   }
   message MyMapOk {
   repeated MapEntryOk map = 1;
   }
   message MyEntityOk {
   optional string name = 1;
   optional MyMapOk keyvalues = 2;
   }

3) Regarding Spark, do you know if anyone is working on implementing these ? I couldn't find a ticket.

[asfimport]

Ryan Blue / @rdblue:
The solution to #1 should be that SparkSQL implements the proposed compatibility rules from this ticket. We're working on moving all of the object models to use those rules and Hive was the first one to do it. You can write the older structures to get something working now, but I don't recommend it as a long-term solution.

For maps, I'm not sure how protobuf encodes them. I didn't think protobuf had a special map type, but relied instead on the caller using a repeated key-value group instead (as you have here). There are currently no rules for encoding a structure in protobuf that shows up as a map in the other object models. To do that, we would either need to add a special case for the name (not ideal) or find a way to signal to parquet-protobuf that you want an annotated map structure. I prefer the second option, but I could be convinced to do the first if there really is no way around it. This should be done in a follow-up ticket so that the other rules aren't blocked.

For Spark other than SparkSQL, I think people mostly use the parquet-mr object models via the newAPIHadoopRDD method. That's not ideal with Avro records, but hopefully we fix that soon enough.

[asfimport]

Ryan Blue / @rdblue:
Issue resolved by pull request 17
https://github.com/apache/incubator-parquet-format/pull/17