Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

standlone模式 oracle到hive1，有时候直接成功 ，有时候失败，经常找不到hive sink #1166

Closed
3 tasks done
LeonYoah opened this issue Aug 19, 2022 · 6 comments · Fixed by #1173
Closed
3 tasks done

standlone模式 oracle到hive1，有时候直接成功 ，有时候失败，经常找不到hive sink #1166

LeonYoah opened this issue Aug 19, 2022 · 6 comments · Fixed by #1173
Labels
bug Something isn't working

Comments

@LeonYoah
Copy link

Search before asking

  • I had searched in the issues and found no similar issues.

What happened

jdk: java version "1.8.0_212
chunjun包是我git clone gitee地址的源码 然后编译 ，然后解压target目录下的 chunjun-assembly-1.12-SNAPSHOT-chunjun-dist.tar.gz 然后解压后吧名字改成 chujun-dist , 他里面有这些 bin ，lib等等 ，然后配置环境变量 为 /opt/module/chunjun-dist
,开始执行local模式，然后报错找不到lib， 发现是他自动指向的lib路径不对，

/usr/java/jdk1.8.0_212/bin/java -cp /../lib/* com.dtstack.chunjun.client.Launcher -job /opt/module/chunjun2/job/oracle_to_hive_test2_insert.json -mode standalone -jobType sync -chunjunDistDir -flinkConfDir /opt/module/flink-1.12.7/conf -flinkLibDir /opt/module/flink-1.12.7/lib
错误: 找不到或无法加载主类 ....lib.binfmt.d

发现他自动指向的lib是上一次目录，然后我就在chunjun-dist目录之上建立一个chunjun目录，该目录下我放了bin,lib和chunuun-dist
修改环境变量为/opt/moudle/chunjun/chujun-dist ,然后在chunjun目录下执行
sh ./bin/chunjun-standalone.sh -job /opt/module/chunjun/job/oracle_to_hive_test2_insert.json

成功了 然后就出现这个问题了

Caused by: org.apache.flink.streaming.runtime.tasks.StreamTaskException: Cannot load user class: com.dtstack.chunjun.connector.hive.sink.HiveOutputFormat
ClassLoader info: URL ClassLoader:
Class not resolvable through given classloader

不仅如此，我试了编译后的chunjun目录，也出现同样的问题

What you expected to happen

2022-08-19 10:28:17.587 [flink-akka.actor.default-dispatcher-1009] INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Received JobGraph submission 2e7d1eb4efd2636364f74213e24baf26 (Flink_Job).
2022-08-19 10:28:17.587 [flink-akka.actor.default-dispatcher-1009] INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Submitting job 2e7d1eb4efd2636364f74213e24baf26 (Flink_Job).
2022-08-19 10:28:17.591 [cluster-io-thread-4] INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.jobmaster.JobMaster at akka://flink/user/rpc/jobmanager_5 .
2022-08-19 10:28:17.592 [cluster-io-thread-4] INFO org.apache.flink.runtime.jobmaster.JobMaster - Initializing job Flink_Job (2e7d1eb4efd2636364f74213e24baf26).
2022-08-19 10:28:17.593 [cluster-io-thread-4] INFO org.apache.flink.runtime.jobmaster.JobMaster - Using restart back off time strategy NoRestartBackoffTimeStrategy for Flink_Job (2e7d1eb4efd2636364f74213e24baf26).
2022-08-19 10:28:17.593 [cluster-io-thread-4] INFO org.apache.flink.runtime.jobmaster.JobMaster - start to buildGraph for job Flink_Job (2e7d1eb4efd2636364f74213e24baf26).
2022-08-19 10:28:17.600 [cluster-io-thread-4] INFO org.apache.flink.runtime.jobmaster.JobMaster - trying to download shipFile from blobServer for job Flink_Job (2e7d1eb4efd2636364f74213e24baf26).
2022-08-19 10:28:17.601 [cluster-io-thread-4] INFO org.apache.flink.runtime.jobmaster.JobMaster - Running initialization on master for job Flink_Job (2e7d1eb4efd2636364f74213e24baf26).
2022-08-19 10:28:17.679 [cluster-io-thread-4] INFO com.dtstack.chunjun.connector.hive.sink.HiveOutputFormat - timeZone = sun.util.calendar.ZoneInfo[id="Asia/Shanghai",offset=28800000,dstSavings=0,useDaylight=false,transitions=29,lastRule=null]
2022-08-19 10:28:17.714 [cluster-io-thread-4] INFO org.apache.flink.runtime.jobmaster.JobMaster - Successfully ran initialization on master in 113 ms.
2022-08-19 10:28:17.784 [cluster-io-thread-4] INFO c.dtstack.chunjun.connector.oracle.source.OracleInputFormat - createInputSplitsInternal success, splits is [
{
"mod": 0,
"isPolling": false,
"splitStrategy": "mod",
"rangeEndLocationOperator": " < ",
"partitionNumber": 0,
"totalNumberOfPartitions": 1
}
]
2022-08-19 10:28:17.785 [cluster-io-thread-4] INFO o.a.flink.runtime.scheduler.adapter.DefaultExecutionTopology - Built 1 pipelined regions in 0 ms
2022-08-19 10:28:17.786 [cluster-io-thread-4] INFO org.apache.flink.runtime.jobmaster.JobMaster - No state backend has been configured, using default (Memory / JobManager) MemoryStateBackend (data in heap memory / checkpoints to JobManager) (checkpoints: 'null', savepoints: 'null', asynchronous: TRUE, maxStateSize: 5242880)
2022-08-19 10:28:17.787 [cluster-io-thread-4] INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - No checkpoint found during restore.
2022-08-19 10:28:17.787 [cluster-io-thread-4] INFO org.apache.flink.runtime.jobmaster.JobMaster - Using failover strategy org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy@26ff5a2e for Flink_Job (2e7d1eb4efd2636364f74213e24baf26).
2022-08-19 10:28:17.787 [cluster-io-thread-4] INFO org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl - JobManager runner for job Flink_Job (2e7d1eb4efd2636364f74213e24baf26) was granted leadership with session id 00000000-0000-0000-0000-000000000000 at akka.tcp://flink@192.168.1.166:6123/user/rpc/jobmanager_5.
2022-08-19 10:28:17.788 [flink-akka.actor.default-dispatcher-1009] INFO org.apache.flink.runtime.jobmaster.JobMaster - Starting execution of job Flink_Job (2e7d1eb4efd2636364f74213e24baf26) under job master id 00000000000000000000000000000000.
2022-08-19 10:28:17.789 [flink-akka.actor.default-dispatcher-1009] INFO org.apache.flink.runtime.jobmaster.JobMaster - Starting scheduling with scheduling strategy [org.apache.flink.runtime.scheduler.strategy.PipelinedRegionSchedulingStrategy]
2022-08-19 10:28:17.789 [flink-akka.actor.default-dispatcher-1009] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job Flink_Job (2e7d1eb4efd2636364f74213e24baf26) switched from state CREATED to RUNNING.
2022-08-19 10:28:17.790 [flink-akka.actor.default-dispatcher-1009] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: oraclesourcefactory (1/1) (358a30f63a48e9a4816a6a1b5681b968) switched from CREATED to SCHEDULED.
2022-08-19 10:28:17.790 [flink-akka.actor.default-dispatcher-1009] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: hivesinkfactory (1/1) (97c72f1302943a417c84bb8188272a16) switched from CREATED to SCHEDULED.
2022-08-19 10:28:17.791 [flink-akka.actor.default-dispatcher-1009] INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot serve slot request, no ResourceManager connected. Adding as pending request [SlotRequestId{d54de734b6d4a10ac346a92602f89bc9}]
2022-08-19 10:28:17.792 [flink-akka.actor.default-dispatcher-1009] INFO org.apache.flink.runtime.jobmaster.JobMaster - Connecting to ResourceManager akka.tcp://flink@192.168.1.166:6123/user/rpc/resourcemanager_*(00000000000000000000000000000000)
2022-08-19 10:28:17.793 [flink-akka.actor.default-dispatcher-1012] INFO org.apache.flink.runtime.jobmaster.JobMaster - Resolved ResourceManager address, beginning registration
2022-08-19 10:28:17.794 [flink-akka.actor.default-dispatcher-1009] INFO o.a.flink.runtime.resourcemanager.StandaloneResourceManager - Registering job manager 00000000000000000000000000000000@akka.tcp://flink@192.168.1.166:6123/user/rpc/jobmanager_5 for job 2e7d1eb4efd2636364f74213e24baf26.
2022-08-19 10:28:17.796 [flink-akka.actor.default-dispatcher-1007] INFO o.a.flink.runtime.resourcemanager.StandaloneResourceManager - Registered job manager 00000000000000000000000000000000@akka.tcp://flink@192.168.1.166:6123/user/rpc/jobmanager_5 for job 2e7d1eb4efd2636364f74213e24baf26.
2022-08-19 10:28:17.797 [flink-akka.actor.default-dispatcher-1009] INFO org.apache.flink.runtime.jobmaster.JobMaster - JobManager successfully registered at ResourceManager, leader id: 00000000000000000000000000000000.
2022-08-19 10:28:17.797 [flink-akka.actor.default-dispatcher-1009] INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Requesting new slot [SlotRequestId{d54de734b6d4a10ac346a92602f89bc9}] and profile ResourceProfile{UNKNOWN} with allocation id 9eb8fa0ebb0c69b7b196ae2f74175c8e from resource manager.
2022-08-19 10:28:17.797 [flink-akka.actor.default-dispatcher-1007] INFO o.a.flink.runtime.resourcemanager.StandaloneResourceManager - Request slot with profile ResourceProfile{UNKNOWN} for job 2e7d1eb4efd2636364f74213e24baf26 with allocation id 9eb8fa0ebb0c69b7b196ae2f74175c8e.
2022-08-19 10:28:17.846 [flink-akka.actor.default-dispatcher-1012] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: oraclesourcefactory (1/1) (358a30f63a48e9a4816a6a1b5681b968) switched from SCHEDULED to DEPLOYING.
2022-08-19 10:28:17.846 [flink-akka.actor.default-dispatcher-1012] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: oraclesourcefactory (1/1) (attempt #0) with attempt id 358a30f63a48e9a4816a6a1b5681b968 to 192.168.1.167:46197-be6535 @ 192.168.1.167 (dataPort=44947) with allocation id 9eb8fa0ebb0c69b7b196ae2f74175c8e
2022-08-19 10:28:17.876 [flink-akka.actor.default-dispatcher-1012] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: hivesinkfactory (1/1) (97c72f1302943a417c84bb8188272a16) switched from SCHEDULED to DEPLOYING.
2022-08-19 10:28:17.876 [flink-akka.actor.default-dispatcher-1012] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Sink: hivesinkfactory (1/1) (attempt #0) with attempt id 97c72f1302943a417c84bb8188272a16 to 192.168.1.167:46197-be6535 @ 192.168.1.167 (dataPort=44947) with allocation id 9eb8fa0ebb0c69b7b196ae2f74175c8e
2022-08-19 10:28:18.111 [flink-akka.actor.default-dispatcher-1012] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: hivesinkfactory (1/1) (97c72f1302943a417c84bb8188272a16) switched from DEPLOYING to RUNNING.
2022-08-19 10:28:18.121 [flink-akka.actor.default-dispatcher-1012] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: oraclesourcefactory (1/1) (358a30f63a48e9a4816a6a1b5681b968) switched from DEPLOYING to RUNNING.
2022-08-19 10:28:18.293 [flink-akka.actor.default-dispatcher-1009] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Sink: hivesinkfactory (1/1) (97c72f1302943a417c84bb8188272a16) switched from RUNNING to FAILED on 192.168.1.167:46197-be6535 @ 192.168.1.167 (dataPort=44947).
org.apache.flink.streaming.runtime.tasks.StreamTaskException: Cannot load user class: com.dtstack.chunjun.connector.hive.sink.HiveOutputFormat
ClassLoader info: URL ClassLoader:
Class not resolvable through given classloader.
at org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperatorFactory(StreamConfig.java:331)
at org.apache.flink.streaming.runtime.tasks.OperatorChain.(OperatorChain.java:150)
at org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:519)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:575)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:758)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:573)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: com.dtstack.chunjun.connector.hive.sink.HiveOutputFormat
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at org.apache.flink.util.FlinkUserCodeClassLoader.loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:64)
at org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:74)
at org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.loadClass(FlinkUserCodeClassLoaders.java:172)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.flink.util.InstantiationUtil$ClassLoaderObjectInputStream.resolveClass(InstantiationUtil.java:76)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1868)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1751)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2042)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:615)
at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:600)
at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:587)
at org.apache.flink.util.InstantiationUtil.readObjectFromConfig(InstantiationUtil.java:541)
at org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperatorFactory(StreamConfig.java:317)
... 6 common frames omitted
2022-08-19 10:28:18.295 [flink-akka.actor.default-dispatcher-1009] INFO o.a.f.r.e.f.flip1.RestartPipelinedRegionFailoverStrategy - Calculating tasks to restart to recover the failed task 0a448493b4782967b150582570326227_0.
2022-08-19 10:28:18.296 [flink-akka.actor.default-dispatcher-1009] INFO o.a.f.r.e.f.flip1.RestartPipelinedRegionFailoverStrategy - 2 tasks should be restarted to recover the failed task 0a448493b4782967b150582570326227_0.
2022-08-19 10:28:18.296 [flink-akka.actor.default-dispatcher-1009] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job Flink_Job (2e7d1eb4efd2636364f74213e24baf26) switched from state RUNNING to FAILING.
org.apache.flink.runtime.JobException: Recovery is suppressed by NoRestartBackoffTimeStrategy
at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:118)
at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:80)
at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:233)
at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:224)
at org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:215)
at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:666)
at org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:89)
at org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:446)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:305)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:212)
at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:158)
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
at scala.PartialFunction.applyOrElse(PartialFunction.scala:123)
at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)
at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
at akka.actor.Actor.aroundReceive(Actor.scala:517)
at akka.actor.Actor.aroundReceive$(Actor.scala:515)
at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
at akka.actor.ActorCell.invoke(ActorCell.scala:561)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
at akka.dispatch.Mailbox.run(Mailbox.scala:225)
at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: org.apache.flink.streaming.runtime.tasks.StreamTaskException: Cannot load user class: com.dtstack.chunjun.connector.hive.sink.HiveOutputFormat
ClassLoader info: URL ClassLoader:
Class not resolvable through given classloader.
at org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperatorFactory(StreamConfig.java:331)
at org.apache.flink.streaming.runtime.tasks.OperatorChain.(OperatorChain.java:150)
at org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:519)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:575)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:758)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:573)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: com.dtstack.chunjun.connector.hive.sink.HiveOutputFormat
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at org.apache.flink.util.FlinkUserCodeClassLoader.loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:64)
at org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:74)
at org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.loadClass(FlinkUserCodeClassLoaders.java:172)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.flink.util.InstantiationUtil$ClassLoaderObjectInputStream.resolveClass(InstantiationUtil.java:76)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1868)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1751)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2042)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:615)
at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:600)
at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:587)
at org.apache.flink.util.InstantiationUtil.readObjectFromConfig(InstantiationUtil.java:541)
at org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperatorFactory(StreamConfig.java:317)
... 6 common frames omitted
2022-08-19 10:28:18.300 [flink-akka.actor.default-dispatcher-1009] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: oraclesourcefactory (1/1) (358a30f63a48e9a4816a6a1b5681b968) switched from RUNNING to CANCELING.
2022-08-19 10:28:18.415 [flink-akka.actor.default-dispatcher-1012] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: oraclesourcefactory (1/1) (358a30f63a48e9a4816a6a1b5681b968) switched from CANCELING to CANCELED.
2022-08-19 10:28:18.417 [flink-akka.actor.default-dispatcher-1012] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job Flink_Job (2e7d1eb4efd2636364f74213e24baf26) switched from state FAILING to FAILED.
org.apache.flink.runtime.JobException: Recovery is suppressed by NoRestartBackoffTimeStrategy
at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:118)
at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:80)
at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:233)
at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:224)
at org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:215)
at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:666)
at org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:89)
at org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:446)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:305)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:212)
at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:158)
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
at scala.PartialFunction.applyOrElse(PartialFunction.scala:123)
at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)
at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
at akka.actor.Actor.aroundReceive(Actor.scala:517)
at akka.actor.Actor.aroundReceive$(Actor.scala:515)
at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
at akka.actor.ActorCell.invoke(ActorCell.scala:561)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
at akka.dispatch.Mailbox.run(Mailbox.scala:225)
at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: org.apache.flink.streaming.runtime.tasks.StreamTaskException: Cannot load user class: com.dtstack.chunjun.connector.hive.sink.HiveOutputFormat
ClassLoader info: URL ClassLoader:
Class not resolvable through given classloader.
at org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperatorFactory(StreamConfig.java:331)
at org.apache.flink.streaming.runtime.tasks.OperatorChain.(OperatorChain.java:150)
at org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:519)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:575)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:758)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:573)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: com.dtstack.chunjun.connector.hive.sink.HiveOutputFormat
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at org.apache.flink.util.FlinkUserCodeClassLoader.loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:64)
at org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:74)
at org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.loadClass(FlinkUserCodeClassLoaders.java:172)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.flink.util.InstantiationUtil$ClassLoaderObjectInputStream.resolveClass(InstantiationUtil.java:76)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1868)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1751)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2042)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:615)
at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:600)
at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:587)
at org.apache.flink.util.InstantiationUtil.readObjectFromConfig(InstantiationUtil.java:541)
at org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperatorFactory(StreamConfig.java:317)
... 6 common frames omitted
2022-08-19 10:28:18.421 [flink-akka.actor.default-dispatcher-1012] INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Stopping checkpoint coordinator for job 2e7d1eb4efd2636364f74213e24baf26.
2022-08-19 10:28:18.422 [flink-akka.actor.default-dispatcher-1012] INFO o.a.f.runtime.checkpoint.StandaloneCompletedCheckpointStore - Shutting down

How to reproduce

oracle 到hive1.x

使用chunjun-assembly-1.12-SNAPSHOT-chunjun-dist.tar.gz这个包|

或者使用编译后的chunjun目录执行 standlone模式 ，oracle到hive1

Anything else

No response

Version

master

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct
@LeonYoah LeonYoah added the bug Something isn't working label Aug 19, 2022
@Huochair
Copy link

E87E8BE6-A527-45BB-8826-387B9388EF60
mysql to mysql 调用，脚本有成功执行的记录

@chestnutqiang
Copy link
Contributor

读写文件，你可以用 hdfswriter。 把目录配置成 hive 表的 location 路径。

@LeonYoah
Copy link
Author

我试过，一样的 也是偶尔找不到hdfs output

@Paddy0523 Paddy0523 linked a pull request Aug 21, 2022 that will close this issue
[hotfix-#1172][assembly] Rebuild the chunjun-dist.tar.gz file structure #1173
Merged
9 tasks
@Paddy0523
Copy link
Contributor

可以把chunjun-dist放到所有节点的相同目录下来解决这个问题。

@LeonYoah
Copy link
Author

同目录下来解决这个问题。

明天试试

@qijizong
Copy link

是的，分发到各节点可以解决

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[hotfix-#1172][assembly] Rebuild the chunjun-dist.tar.gz file structure Paddy0523/chunjun
5 participants
@chestnutqiang @qijizong @Huochair @Paddy0523 @LeonYoah