You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository has been archived by the owner on Feb 7, 2020. It is now read-only.
I just ran through the installation instructions with the docker container. While the example in the instructions for basic_http worked, basic_summary fails as follows:
Executing pig -v -x local -f pig/examples/basic_summary.pig -param pcap=data/web.pcap -param output=output -param n=1 -param snortconfig=lib/snort/etc/snort.conf -param tcppath=lib/scripts/tcp.py -param dnspath=lib/scripts/dns_parser.py
2014-04-14 00:31:03,151 [main] INFO org.apache.pig.Main - Logging error messages to: /src/packetpig/pig_1397435463143.log
2014-04-14 00:31:03,245 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:///
2014-04-14 00:31:03,472 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: UNKNOWN
2014-04-14 00:31:03,472 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - pig.usenewlogicalplan is set to true. New logical plan will be used.
2014-04-14 00:31:03,521 [main] INFO org.apache.pig.newplan.logical.rules.ColumnPruneVisitor - Columns pruned for packets: $0, $1, $2, $3, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26, $27, $28, $29, $30, $31, $32
2014-04-14 00:31:03,567 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - (Name: summary: Store(file:/tmp/temp-1405130183/tmp-2050501636:org.apache.pig.impl.io.InterStorage) - scope-6 Operator Key: scope-6)
2014-04-14 00:31:03,573 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2014-04-14 00:31:03,587 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2014-04-14 00:31:03,587 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2014-04-14 00:31:03,632 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics with processName=JobTracker, sessionId=
2014-04-14 00:31:03,639 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2014-04-14 00:31:03,656 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2014-04-14 00:31:05,902 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2014-04-14 00:31:05,925 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2014-04-14 00:31:05,925 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
2014-04-14 00:31:05,934 [Thread-1] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2014-04-14 00:31:06,052 [Thread-1] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
2014-04-14 00:31:06,054 [Thread-1] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1
2014-04-14 00:31:06,251 [Thread-12] INFO org.apache.hadoop.mapred.LocalJobRunner - Waiting for map tasks
2014-04-14 00:31:06,253 [pool-1-thread-1] INFO org.apache.hadoop.mapred.LocalJobRunner - Starting task: attempt_local_0001_m_000000_0
2014-04-14 00:31:06,296 [pool-1-thread-1] INFO org.apache.hadoop.util.ProcessTree - setsid exited with exit code 0
2014-04-14 00:31:06,302 [pool-1-thread-1] INFO org.apache.hadoop.mapred.Task - Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@64498713
2014-04-14 00:31:06,306 [pool-1-thread-1] INFO org.apache.hadoop.mapred.MapTask - Processing split: Number of splits :1
Total Length = 358584
Input split[0]:
Length = 358584
Locations:
2014-04-14 00:31:06,364 [Thread-12] INFO org.apache.hadoop.mapred.LocalJobRunner - Map task executor complete.
2014-04-14 00:31:06,365 [Thread-12] WARN org.apache.hadoop.mapred.LocalJobRunner - job_local_0001
java.lang.Exception: org.apache.pig.backend.executionengine.ExecException: ERROR 2106: Error while computing count in COUNT
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:349)
Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2106: Error while computing count in COUNT
at org.apache.pig.builtin.COUNT.exec(COUNT.java:71)
at org.apache.pig.builtin.COUNT.exec(COUNT.java:51)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:245)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:322)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:360)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:290)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:237)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:232)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:53)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:648)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Caused by: java.lang.ClassCastException: java.lang.Integer cannot be cast to org.apache.pig.data.DataBag
at org.apache.pig.builtin.COUNT.exec(COUNT.java:57)
... 18 more
2014-04-14 00:31:06,426 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_local_0001
2014-04-14 00:31:06,426 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
2014-04-14 00:31:11,427 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - job job_local_0001 has failed! Stop running all dependent jobs
2014-04-14 00:31:11,431 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
2014-04-14 00:31:11,432 [main] ERROR org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
2014-04-14 00:31:11,432 [main] INFO org.apache.pig.tools.pigstats.PigStats - Detected Local mode. Stats reported below may be incomplete
2014-04-14 00:31:11,433 [main] INFO org.apache.pig.tools.pigstats.PigStats - Script Statistics:
Failed Jobs:
JobId Alias Feature Message Outputs
job_local_0001 packets,summary MAP_ONLY Message: Job failed! Error - NA file:/tmp/temp-1405130183/tmp-2050501636,
Input(s):
Failed to read data from "file:///src/packetpig/data/web.pcap"
Output(s):
Failed to produce result in "file:/tmp/temp-1405130183/tmp-2050501636"
Job DAG:
job_local_0001
2014-04-14 00:31:11,433 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
2014-04-14 00:31:11,436 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1066: Unable to open iterator for alias summary
2014-04-14 00:31:11,436 [main] ERROR org.apache.pig.tools.grunt.Grunt - org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias summary
at org.apache.pig.PigServer.openIterator(PigServer.java:765)
at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:644)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:171)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:147)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:90)
at org.apache.pig.Main.run(Main.java:396)
at org.apache.pig.Main.main(Main.java:107)
Caused by: java.io.IOException: Job terminated with anomalous status FAILED
at org.apache.pig.PigServer.openIterator(PigServer.java:755)
... 7 more
Details also at logfile: /src/packetpig/pig_1397435463143.log
ERROR 1066: Unable to open iterator for alias summary
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias summary
at org.apache.pig.PigServer.openIterator(PigServer.java:765)
at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:644)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:171)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:147)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:90)
at org.apache.pig.Main.run(Main.java:396)
at org.apache.pig.Main.main(Main.java:107)
Caused by: java.io.IOException: Job terminated with anomalous status FAILED
at org.apache.pig.PigServer.openIterator(PigServer.java:755)
... 7 more
The text was updated successfully, but these errors were encountered:
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.
I just ran through the installation instructions with the docker container. While the example in the instructions for basic_http worked, basic_summary fails as follows:
root@2390b772f892:/src/packetpig# ./pigrun.py -f pig/examples/basic_summary.pig
['pig -v',
'-x local',
'-f pig/examples/basic_summary.pig',
'-param pcap=data/web.pcap',
'-param output=output',
'-param n=1',
'-param snortconfig=lib/snort/etc/snort.conf',
'-param tcppath=lib/scripts/tcp.py',
'-param dnspath=lib/scripts/dns_parser.py']
Executing pig -v -x local -f pig/examples/basic_summary.pig -param pcap=data/web.pcap -param output=output -param n=1 -param snortconfig=lib/snort/etc/snort.conf -param tcppath=lib/scripts/tcp.py -param dnspath=lib/scripts/dns_parser.py
2014-04-14 00:31:03,151 [main] INFO org.apache.pig.Main - Logging error messages to: /src/packetpig/pig_1397435463143.log
2014-04-14 00:31:03,245 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:///
2014-04-14 00:31:03,472 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: UNKNOWN
2014-04-14 00:31:03,472 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - pig.usenewlogicalplan is set to true. New logical plan will be used.
2014-04-14 00:31:03,521 [main] INFO org.apache.pig.newplan.logical.rules.ColumnPruneVisitor - Columns pruned for packets: $0, $1, $2, $3, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26, $27, $28, $29, $30, $31, $32
2014-04-14 00:31:03,567 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - (Name: summary: Store(file:/tmp/temp-1405130183/tmp-2050501636:org.apache.pig.impl.io.InterStorage) - scope-6 Operator Key: scope-6)
2014-04-14 00:31:03,573 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2014-04-14 00:31:03,587 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2014-04-14 00:31:03,587 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2014-04-14 00:31:03,632 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics with processName=JobTracker, sessionId=
2014-04-14 00:31:03,639 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2014-04-14 00:31:03,656 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2014-04-14 00:31:05,902 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2014-04-14 00:31:05,925 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2014-04-14 00:31:05,925 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
2014-04-14 00:31:05,934 [Thread-1] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2014-04-14 00:31:06,052 [Thread-1] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
2014-04-14 00:31:06,054 [Thread-1] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1
2014-04-14 00:31:06,251 [Thread-12] INFO org.apache.hadoop.mapred.LocalJobRunner - Waiting for map tasks
2014-04-14 00:31:06,253 [pool-1-thread-1] INFO org.apache.hadoop.mapred.LocalJobRunner - Starting task: attempt_local_0001_m_000000_0
2014-04-14 00:31:06,296 [pool-1-thread-1] INFO org.apache.hadoop.util.ProcessTree - setsid exited with exit code 0
2014-04-14 00:31:06,302 [pool-1-thread-1] INFO org.apache.hadoop.mapred.Task - Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@64498713
2014-04-14 00:31:06,306 [pool-1-thread-1] INFO org.apache.hadoop.mapred.MapTask - Processing split: Number of splits :1
Total Length = 358584
Input split[0]:
Length = 358584
Locations:
2014-04-14 00:31:06,364 [Thread-12] INFO org.apache.hadoop.mapred.LocalJobRunner - Map task executor complete.
2014-04-14 00:31:06,365 [Thread-12] WARN org.apache.hadoop.mapred.LocalJobRunner - job_local_0001
java.lang.Exception: org.apache.pig.backend.executionengine.ExecException: ERROR 2106: Error while computing count in COUNT
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:349)
Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2106: Error while computing count in COUNT
at org.apache.pig.builtin.COUNT.exec(COUNT.java:71)
at org.apache.pig.builtin.COUNT.exec(COUNT.java:51)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:245)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:322)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:360)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:290)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:237)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:232)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:53)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:648)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Caused by: java.lang.ClassCastException: java.lang.Integer cannot be cast to org.apache.pig.data.DataBag
at org.apache.pig.builtin.COUNT.exec(COUNT.java:57)
... 18 more
2014-04-14 00:31:06,426 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_local_0001
2014-04-14 00:31:06,426 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
2014-04-14 00:31:11,427 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - job job_local_0001 has failed! Stop running all dependent jobs
2014-04-14 00:31:11,431 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
2014-04-14 00:31:11,432 [main] ERROR org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
2014-04-14 00:31:11,432 [main] INFO org.apache.pig.tools.pigstats.PigStats - Detected Local mode. Stats reported below may be incomplete
2014-04-14 00:31:11,433 [main] INFO org.apache.pig.tools.pigstats.PigStats - Script Statistics:
HadoopVersion PigVersion UserId StartedAt FinishedAt Features
0.20.2-cdh3u6 0.8.1-cdh3u6 root 2014-04-14 00:31:03 2014-04-14 00:31:11 UNKNOWN
Failed!
Failed Jobs:
JobId Alias Feature Message Outputs
job_local_0001 packets,summary MAP_ONLY Message: Job failed! Error - NA file:/tmp/temp-1405130183/tmp-2050501636,
Input(s):
Failed to read data from "file:///src/packetpig/data/web.pcap"
Output(s):
Failed to produce result in "file:/tmp/temp-1405130183/tmp-2050501636"
Job DAG:
job_local_0001
2014-04-14 00:31:11,433 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
2014-04-14 00:31:11,436 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1066: Unable to open iterator for alias summary
2014-04-14 00:31:11,436 [main] ERROR org.apache.pig.tools.grunt.Grunt - org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias summary
at org.apache.pig.PigServer.openIterator(PigServer.java:765)
at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:644)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:171)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:147)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:90)
at org.apache.pig.Main.run(Main.java:396)
at org.apache.pig.Main.main(Main.java:107)
Caused by: java.io.IOException: Job terminated with anomalous status FAILED
at org.apache.pig.PigServer.openIterator(PigServer.java:755)
... 7 more
Details also at logfile: /src/packetpig/pig_1397435463143.log
root@2390b772f892:/src/packetpig# cat pig_1397435463143.log
Pig Stack Trace
ERROR 1066: Unable to open iterator for alias summary
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias summary
at org.apache.pig.PigServer.openIterator(PigServer.java:765)
at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:644)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:171)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:147)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:90)
at org.apache.pig.Main.run(Main.java:396)
at org.apache.pig.Main.main(Main.java:107)
Caused by: java.io.IOException: Job terminated with anomalous status FAILED
at org.apache.pig.PigServer.openIterator(PigServer.java:755)
... 7 more
The text was updated successfully, but these errors were encountered: