NPE when reading metadata for manifests.

[chenjunjiedada]

When using Spark3SQL to read manifests, it throws NPE exception. The stack is：
Caused by: java.lang.NullPointerException
at org.apache.iceberg.spark.source.StructInternalRow.getBoolean(StructInternalRow.java:109)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.iceberg.common.DynMethods$UnboundMethod.invokeChecked(DynMethods.java:65)
at org.apache.iceberg.common.DynMethods$UnboundMethod.invoke(DynMethods.java:77)
at org.apache.iceberg.common.DynMethods$BoundMethod.invoke(DynMethods.java:180)
at org.apache.iceberg.spark.source.RowDataReader.lambda$newDataIterable$3(RowDataReader.java:197)
at org.apache.iceberg.io.CloseableIterable$4$1.next(CloseableIterable.java:114)
at org.apache.iceberg.io.FilterIterator.advance(FilterIterator.java:66)
at org.apache.iceberg.io.FilterIterator.hasNext(FilterIterator.java:50)
at org.apache.iceberg.spark.source.BaseDataReader.next(BaseDataReader.java:87)
at org.apache.spark.sql.execution.datasources.v2.PartitionIterator.hasNext(DataSourceRDD.scala:79)
at org.apache.spark.sql.execution.datasources.v2.MetricsIterator.hasNext(DataSourceRDD.scala:112)
at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:729)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:221)
at org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:299)
at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$1(BlockManager.scala:1371)
at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1298)
at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1362)
at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1186)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:360)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:311)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:127)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1391)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
... 3 more

[RussellSpitzer]

Do you have any more info on the repo for this?

[chenjunjiedada]

Thanks @RussellSpitzer to take care! It happens at the customer's table. I opened the issue to remind myself as I was busy. Let me inspect what's happened!

[chenjunjiedada]

I thought it is caused by the ManifestFileUtil bug. But we had fixed that issue internally. After digging more, we found this is caused by our internal flink iceberg sink which appends manifest without partition summary.

[chenjunjiedada]

Still hit this problem. Now I know how to reproduce this.

1. Write the iceberg table with the 0.10.0 version.
2. Read the iceberg table manifests with the 0.11.1 version.

ManifestTable.partitionSummarirsToRows puts summary.containsNaN(), which is null, to the spark row that will be parsed to Boolean class and show out.

The following code could fix the issue but I think we should consider with #2495 as well:

--- a/api/src/main/java/org/apache/iceberg/ManifestFile.java
+++ b/api/src/main/java/org/apache/iceberg/ManifestFile.java
@@ -207,7 +207,7 @@ public interface ManifestFile {
      * Default to return null to ensure backward compatibility.
      */
     default Boolean containsNaN() {
-      return null;
+      return true;
     }
 
     /**
--- a/core/src/main/java/org/apache/iceberg/GenericPartitionFieldSummary.java
+++ b/core/src/main/java/org/apache/iceberg/GenericPartitionFieldSummary.java
@@ -121,7 +121,7 @@ public class GenericPartitionFieldSummary
 
   @Override
   public Boolean containsNaN() {
-    return containsNaN;
+    return containsNaN == null || containsNull;
   }

@RussellSpitzer @yyanyy

[RussellSpitzer]

I think I have this fixed in #2495

…

Sent from my iPhone

On Apr 25, 2021, at 3:24 AM, Chen, Junjie ***@***.***> wrote:  Still hit this problem. Now I know how to reproduce this. Write the iceberg table with the 0.10.0 version. Read the iceberg table manifests with the 0.11.1 version. The following code could fix the issue: --- a/core/src/main/java/org/apache/iceberg/GenericPartitionFieldSummary.java +++ b/core/src/main/java/org/apache/iceberg/GenericPartitionFieldSummary.java @@ -121,7 +121,7 @@ public class GenericPartitionFieldSummary @OverRide public Boolean containsNaN() { - return containsNaN; + return containsNaN == null || containsNull; } @RussellSpitzer @yyanyy, We may need to re-consider this with #2492 again. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[yyanyy]

I think it's expected that summary.containsNaN() could return null? looking from the code it seems that StaticDataTask.Row should also be able to handle null? when does the issue occur?

[chenjunjiedada]

@RussellSpitzer, summary.containsNaN() is called in two places, #2495 fixes one place. Another place is in #2509.

@yyanyy , I thought not return null in summary.containsNaN() may fix the problem for all places, while I don't want to break the original assumption. There is also a unit test for the null value. This issue happens when reading manifests as a table.

ManifestTable.partitionSummarirsToRows puts summary.containsNaN(), which is null, to the spark row that will be parsed to Boolean class and show out.

[chenjunjiedada]

@yyanyy Looks like StaticDataTask.Row doesn't handle the null case. Do you think we should fix in that?

[RussellSpitzer]

@chenjunjiedada Good catch, we should definitely fix that, I also was a little worried about keeping the boxed boolean as the return value in containsNaN since we usually treat the parameter as a primitive, but I'm ok with just fixing the other occurrence for now, wdyt?

[RussellSpitzer]

I think if we make the changes @chenjunjiedada suggests (which I think are a good idea too), we should probably just change the signature to be a primitive and not the Boxed bool.