求助 Failed to run job : Application application_1591113812497_103797 failed 2 times 作业运行失败 #65
Description
[2020-06-25 22:33:23.064]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=100M; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=200M; support was removed in 8.0
[2020-06-25 22:33:23.065]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=100M; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=200M; support was removed in 8.0
For more detailed output, check the application tracking page: http://ecs-hn1b-bd-cdp-edg-2:8188/applicationhistory/app/application_1591113812497_103797 Then click on links to logs of each attempt.
. Failing the application.
at com.tencent.angel.client.yarn.AngelYarnClient.updateMaster(AngelYarnClient.java:517)
at com.tencent.angel.client.yarn.AngelYarnClient.startPSServer(AngelYarnClient.java:170)
at com.tencent.angel.client.AngelPSClient.startPS(AngelPSClient.java:115)
at com.tencent.angel.sona.core.DriverContext.startAngelAndPSAgent(DriverContext.scala:97)
at com.tencent.angel.sona.examples.JsonRunnerExamples$.main(JsonRunnerExamples.scala:69)
at com.tencent.angel.sona.examples.JsonRunnerExamples.main(JsonRunnerExamples.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:851)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:926)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:935)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Exception in thread "main" com.tencent.angel.exception.AngelException: java.io.IOException: Failed to run job : Application application_1591113812497_103797 failed 2times (global limit =3; local limit is =2) due to AM Container for appattempt_1591113812497_103797_000002 exited with exitCode: 1
Failing this attempt.Diagnostics: [2020-06-25 22:33:23.063]Exception from container-launch.
Container id: container_e74_1591113812497_103797_02_000001
Exit code: 1
[2020-06-25 22:33:23.064]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=100M; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=200M; support was removed in 8.0
[2020-06-25 22:33:23.065]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=100M; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=200M; support was removed in 8.0
For more detailed output, check the application tracking page: http://ecs-hn1b-bd-cdp-edg-2:8188/applicationhistory/app/application_1591113812497_103797 Then click on links to logs of each attempt.
. Failing the application.
at com.tencent.angel.client.yarn.AngelYarnClient.startPSServer(AngelYarnClient.java:176)
at com.tencent.angel.client.AngelPSClient.startPS(AngelPSClient.java:115)
at com.tencent.angel.sona.core.DriverContext.startAngelAndPSAgent(DriverContext.scala:97)
at com.tencent.angel.sona.examples.JsonRunnerExamples$.main(JsonRunnerExamples.scala:69)
at com.tencent.angel.sona.examples.JsonRunnerExamples.main(JsonRunnerExamples.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:851)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:926)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:935)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.io.IOException: Failed to run job : Application application_1591113812497_103797 failed 2 times (global limit =3; local limit is =2) due to AM Container for appattempt_1591113812497_103797_000002 exited with exitCode: 1
Failing this attempt.Diagnostics: [2020-06-25 22:33:23.063]Exception from container-launch.
Container id: container_e74_1591113812497_103797_02_000001
Exit code: 1
[2020-06-25 22:33:23.064]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=100M; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=200M; support was removed in 8.0
[2020-06-25 22:33:23.065]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=100M; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=200M; support was removed in 8.0
For more detailed output, check the application tracking page: http://ecs-hn1b-bd-cdp-edg-2:8188/applicationhistory/app/application_1591113812497_103797 Then click on links to logs of each attempt.
. Failing the application.
at com.tencent.angel.client.yarn.AngelYarnClient.updateMaster(AngelYarnClient.java:517)
at com.tencent.angel.client.yarn.AngelYarnClient.startPSServer(AngelYarnClient.java:170)
... 16 more
20/06/25 22:33:15 INFO spark.SparkContext: Invoking stop() from shutdown hook
20/06/25 22:33:15 INFO server.AbstractConnector: Stopped Spark@7a389761{HTTP/1.1,[http/1.1]}{0.0.0.0:4041}
20/06/25 22:33:15 INFO ui.SparkUI: Stopped Spark web UI at http://ecs-hn1a-xng-alg-rcmd-edg-1:4041
20/06/25 22:33:15 INFO cluster.YarnClientSchedulerBackend: Interrupting monitor thread
20/06/25 22:33:15 INFO cluster.YarnClientSchedulerBackend: Shutting down all executors
20/06/25 22:33:15 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to shut down
20/06/25 22:33:15 INFO cluster.SchedulerExtensionServices: Stopping SchedulerExtensionServices
(serviceOption=None,
services=List(),
started=false)
20/06/25 22:33:15 INFO cluster.YarnClientSchedulerBackend: Stopped
20/06/25 22:33:15 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
20/06/25 22:33:15 INFO memory.MemoryStore: MemoryStore cleared
20/06/25 22:33:15 INFO storage.BlockManager: BlockManager stopped
20/06/25 22:33:15 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
20/06/25 22:33:15 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
20/06/25 22:33:15 INFO spark.SparkContext: Successfully stopped SparkContext
20/06/25 22:33:15 INFO util.ShutdownHookManager: Shutdown hook called
20/06/25 22:33:15 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-a689fb57-937d-4d2b-bf50-f59d08e9b6b8
20/06/25 22:33:15 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-2d255dca-b9e1-45ad-ba9d-6b8beae996e3
20/06/25 22:33:15 INFO client.AngelClient: stop the application
20/06/25 22:33:15 INFO client.AngelClient: master is null, just kill the application
20/06/25 22:33:15 INFO impl.YarnClientImpl: Killed application application_1591113812497_103797