UT TestSummary failed with "java.lang.RuntimeException: Usage: B = FOREACH (GROUP A ALL) GENERATE Summary(A); Can not get schema from null" when Pig >=0.15

[asfimport]

org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable to store alias B
at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1694)
at org.apache.pig.PigServer.registerQuery(PigServer.java:623)
at org.apache.pig.PigServer.registerQuery(PigServer.java:636)
at parquet.pig.summary.TestSummary.testMaxIsZero(TestSummary.java:154)
...
Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 0: java.lang.RuntimeException: Usage: B = FOREACH (GROUP A ALL) GENERATE Summary(A); Can not get schema from null
at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:307)
at org.apache.pig.PigServer.launchPlan(PigServer.java:1390)
at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1375)
at org.apache.pig.PigServer.execute(PigServer.java:1364)
at org.apache.pig.PigServer.access$500(PigServer.java:113)
at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1689)
... 32 more
Caused by: java.lang.RuntimeException: Usage: B = FOREACH (GROUP A ALL) GENERATE Summary(A); Can not get schema from null
at parquet.pig.summary.Summary.setInputSchema(Summary.java:266)
at org.apache.pig.newplan.logical.expression.ExpToPhyTranslationVisitor.visit(ExpToPhyTranslationVisitor.java:530)
at org.apache.pig.newplan.logical.expression.UserFuncExpression.accept(UserFuncExpression.java:132)
at org.apache.pig.newplan.ReverseDependencyOrderWalkerWOSeenChk.walk(ReverseDependencyOrderWalkerWOSeenChk.java:69)
at org.apache.pig.newplan.logical.relational.LogToPhyTranslationVisitor.visit(LogToPhyTranslationVisitor.java:808)
at org.apache.pig.newplan.logical.relational.LOForEach.accept(LOForEach.java:87)
at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52)
at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:258)
at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:293)
... 37 more
Caused by: java.lang.NullPointerException
at parquet.pig.summary.Summary.setInputSchema(Summary.java:261)
... 46 more

It relates to a change on pig side: pig/src/org/apache/pig/newplan/logical/expression/ExpToPhyTranslationVisitor.java introduced by PIG-3294

Reporter: Xiang Li
Assignee: Thomas Friedrich / @tfriedr

Related issues:

• Class Summary does not provide a getter to return inputSchema (is blocked by)
• Allow Pig use Hive UDFs (relates to)

Original Issue Attachments:

• PARQUET-334-1.patch

PRs and other links:

• PR

_{Note: This issue was originally created as PARQUET-334. Please see the migration documentation for further details.}

[asfimport]

Xiang Li:
Update parquet-pig/src/main/java/parquet/pig/summary/Summary.java to yield a more clear stack trace.

java.lang.NullPointerException
at parquet.pig.summary.Summary.setInputSchema(Summary.java:261)
at org.apache.pig.newplan.logical.expression.ExpToPhyTranslationVisitor.visit(ExpToPhyTranslationVisitor.java:512)
at org.apache.pig.newplan.logical.expression.UserFuncExpression.accept(UserFuncExpression.java:113)
at org.apache.pig.newplan.ReverseDependencyOrderWalkerWOSeenChk.walk(ReverseDependencyOrderWalkerWOSeenChk.java:69)
at org.apache.pig.newplan.logical.relational.LogToPhyTranslationVisitor.visit(LogToPhyTranslationVisitor.java:807)
at org.apache.pig.newplan.logical.relational.LOForEach.accept(LOForEach.java:87)
at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52)
at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:260)
at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:295)
at org.apache.pig.PigServer.launchPlan(PigServer.java:1390)
at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1375)
at org.apache.pig.PigServer.execute(PigServer.java:1364)
at org.apache.pig.PigServer.access$500(PigServer.java:113)
at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1689)
at org.apache.pig.PigServer.registerQuery(PigServer.java:623)
at org.apache.pig.PigServer.registerQuery(PigServer.java:636)
at parquet.pig.summary.TestSummary.testPigScript(TestSummary.java:139

[asfimport]

Xiang Li:
In Pig code, src/org/apache/pig/EvalFunc.java. A private number "inputSchemaInternal" represent the schema. Setter and Getter are also provided

316     private Schema inputSchemaInternal=null;

328     /**
329      * This method is for internal use. It is called by Pig core in both front-end
330      * and back-end to setup the right input schema for EvalFunc
331      */
332     public void setInputSchema(Schema input){
333         this.inputSchemaInternal=input;
334     }
335 
336     /**
337      * This method is intended to be called by the user in {@link EvalFunc} to get the input
338      * schema of the EvalFunc
339      */
340     public Schema getInputSchema(){
341         return this.inputSchemaInternal;
342     }

But actually, they are overrided. In parquet-mr/parquet-pig/src/main/java/parquet/pig/summary/Summary.java, It uses a new number called inputSchema(vs. inputSchemaInternal) to represent schema and override setInputSchema(), but not override getInputSchema()

51  public class Summary extends EvalFunc<String> implements Algebraic {

54     private Schema inputSchema;

257   @Override
258   public void setInputSchema(Schema input) {
259     try {
260       // relation.bag.tuple
261       this.inputSchema=input.getField(0).schema.getField(0).schema;
262       saveSchemaToUDFContext();
263     } catch (FrontendException e) {
264       throw new RuntimeException("Usage: B = FOREACH (GROUP A ALL) GENERATE Summary(A); Can not get schema from " + input, e);
265     } catch (RuntimeException e) {
266       throw new RuntimeException("Usage: B = FOREACH (GROUP A ALL) GENERATE Summary(A); Can not get schema from "+input, e);
267     }
268   }

[asfimport]

Daniel Dai / @daijyc:
Input Schema is maintained by Pig inside EvalFunc. No need to maintain this in Parquet side. Attach patch.

[asfimport]

Xiang Li:
Thanks Daniel for taking care of this!
+1 for the patch, more reasonable to fix it on Parquet side. UT passed on Parquet 1.8.0

Hi Julien, could you please give a review?

[asfimport]

Thomas Friedrich / @tfriedr:
I updated the patch from Daniel and removed the private inputSchema variable and instead call the getInputSchema method of the parent class. Otherwise inputSchema was always null.
@julienledem, can you please review my pull-request with the patch.

[asfimport]

Julien Le Dem / @julienledem:
Issue resolved by pull request 292
#292

[asfimport]

Thomas Friedrich / @tfriedr:
Thanks, @julienledem. Shouldn't the fix version be a parquet-mr release, not parquet-format?