Exception in thread "main" java.lang.NoClassDefFoundError: scala/Product$class Error while running cronjobs in Spark

[sravani-64]

Hello team,

We have setup reports for release 4.10.0 version by building and deploying the following jobs

1. AnalyticsCore
   Build - https://github.com/project-sunbird/sunbird-analytics-core.git, release-4.10.5_RC1
   Deploy - https://github.com/project-sunbird/sunbird-data-pipeline.git, release-4.10.5_RC2
2. CoreDataProducts
   Build - https://github.com/project-sunbird/sunbird-core-dataproducts.git, release-4.10.5_RC2
   Deploy - https://github.com/project-sunbird/sunbird-data-pipeline.git, release-4.10.5_RC2
3. EdDataProducts
   Build - https://github.com/Sunbird-Ed/sunbird-data-products.git, release-4.10.5_RC6
   Deploy - https://github.com/project-sunbird/sunbird-data-pipeline.git, release-4.10.5_RC2

As the reports are not generated, we executed cronjob manually and found below errors

3/07/20 17:27:43 INFO CodeGenerator: Code generated in 11.534373 ms
23/07/20 17:27:43 INFO SparkUI: Stopped Spark web UI at http://54.255.154.146:4040
23/07/20 17:27:43 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
23/07/20 17:27:43 INFO MemoryStore: MemoryStore cleared
23/07/20 17:27:43 INFO BlockManager: BlockManager stopped
23/07/20 17:27:43 INFO BlockManagerMaster: BlockManagerMaster stopped
23/07/20 17:27:43 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
23/07/20 17:27:43 INFO SparkContext: Successfully stopped SparkContext
Exception in thread "main" java.lang.NoClassDefFoundError: scala/Product$class
at com.redislabs.provider.redis.RedisEndpoint.(RedisConfig.scala:28)
at com.redislabs.provider.redis.RedisEndpoint.(RedisConfig.scala:36)
at org.apache.spark.sql.redis.RedisSourceRelation.(RedisSourceRelation.scala:36)
at org.apache.spark.sql.redis.DefaultSource.createRelation(DefaultSource.scala:42)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:353)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:325)
at org.apache.spark.sql.DataFrameReader.$anonfun$load$3(DataFrameReader.scala:307)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:307)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:225)
at org.sunbird.analytics.exhaust.BaseReportsJob.loadData(BaseReportsJob.scala:13)
at org.sunbird.analytics.exhaust.BaseReportsJob.loadData$(BaseReportsJob.scala:11)
at org.sunbird.analytics.exhaust.collection.ProgressExhaustJob$.loadData(ProgressExhaustJob.scala:19)
at org.sunbird.analytics.exhaust.collection.BaseCollectionExhaustJob.getUserCacheDF(BaseCollectionExhaustJob.scala:488)
at org.sunbird.analytics.exhaust.collection.BaseCollectionExhaustJob.getUserCacheDF$(BaseCollectionExhaustJob.scala:486)
at org.sunbird.analytics.exhaust.collection.ProgressExhaustJob$.getUserCacheDF(ProgressExhaustJob.scala:19)
at org.sunbird.analytics.exhaust.collection.BaseCollectionExhaustJob.$anonfun$execute$3(BaseCollectionExhaustJob.scala:108)
at org.ekstep.analytics.framework.util.CommonUtil$.time(CommonUtil.scala:538)
at org.sunbird.analytics.exhaust.collection.BaseCollectionExhaustJob.execute(BaseCollectionExhaustJob.scala:107)
at org.sunbird.analytics.exhaust.collection.BaseCollectionExhaustJob.execute$(BaseCollectionExhaustJob.scala:101)
at org.sunbird.analytics.exhaust.collection.ProgressExhaustJob$.execute(ProgressExhaustJob.scala:19)
at org.sunbird.analytics.exhaust.collection.BaseCollectionExhaustJob.$anonfun$main$1(BaseCollectionExhaustJob.scala:67)
at org.ekstep.analytics.framework.util.CommonUtil$.time(CommonUtil.scala:538)
at org.sunbird.analytics.exhaust.collection.BaseCollectionExhaustJob.main(BaseCollectionExhaustJob.scala:67)
at org.sunbird.analytics.exhaust.collection.BaseCollectionExhaustJob.main$(BaseCollectionExhaustJob.scala:57)
at org.sunbird.analytics.exhaust.collection.ProgressExhaustJob$.main(ProgressExhaustJob.scala:19)
at org.ekstep.analytics.job.JobExecutor$.main(JobExecutor.scala:16)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at optional.Application.callWithOptions(application.scala:251)
at optional.Application.callWithOptions$(application.scala:223)
at org.ekstep.analytics.job.JobExecutor$.callWithOptions(JobExecutor.scala:8)
at optional.Application.main(application.scala:258)
at optional.Application.main$(application.scala:255)
at org.ekstep.analytics.job.JobExecutor$.main(JobExecutor.scala:8)
at org.ekstep.analytics.job.JobExecutor.main(JobExecutor.scala)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:951)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1039)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1048)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: scala.Product$class
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:588)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
... 50 more
23/07/20 17:27:43 INFO ShutdownHookManager: Shutdown hook called
23/07/20 17:27:43 INFO ShutdownHookManager: Deleting directory /mount/data/analytics/tmp/spark-3755a92a-0077-4751-ac47-e86c6ed1ee94
23/07/20 17:27:43 INFO ShutdownHookManager: Deleting directory /tmp/spark-c3ce3112-6127-4e1f-8518-6285997ee008
23/07/20 17:27:43 INFO CassandraConnector: Disconnected from Cassandra cluster.
23/07/20 17:27:43 INFO SerialShutdownHooks: Successfully executed shutdown hook: Clearing session cache for C* connector
Job execution completed - progress-exhaust

Can someone pls help in resolving this error. Thank you

[manjudr]

@sravani-64 - What is the version of Java we are using? Have we provisioned Spark properly? It appears to be a version mismatch issue. Please ensure that we have provisioned Spark version 3.1.0 and Java version 11 as required.

[sravani-64]

@manjudr Spark version is 3.1.3 and Java version is 11.0.2

[manjudr]

Have we rebuilt the artifacts, or are we simply reusing the existing artifacts?

[sravani-64]

We rebuilt the artifacts

[manjudr]

Ok - Can we rebuild all the repos below sequence once?

1. Build Analytics Core and Deploy
2. Build Core Data products and Deploy
3. Build Ed Data Products and Deploy

[sravani-64]

@manjudr Successfully built and deployed
Build Analytics Core and Deploy
Build Core Data products and Deploy
Build Ed Data Products and Deploy

But the errors are same as earlier

23/07/21 13:12:29 INFO CodeGenerator: Code generated in 170.910612 ms
23/07/21 13:12:29 INFO SparkContext: Starting job: first at BaseCollectionExhaustJob.scala:440
23/07/21 13:12:29 INFO DAGScheduler: Got job 0 (first at BaseCollectionExhaustJob.scala:440) with 1 output partitions
23/07/21 13:12:29 INFO DAGScheduler: Final stage: ResultStage 0 (first at BaseCollectionExhaustJob.scala:440)
23/07/21 13:12:29 INFO DAGScheduler: Parents of final stage: List()
23/07/21 13:12:29 INFO DAGScheduler: Missing parents: List()
23/07/21 13:12:29 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[3] at first at BaseCollectionExhaustJob.scala:440), which has no missing parents
23/07/21 13:12:29 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 14.7 KiB, free 2.0 GiB)
23/07/21 13:12:29 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 7.4 KiB, free 2.0 GiB)
23/07/21 13:12:29 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on dp-services.internal.cloudapp.net:39351 (size: 7.4 KiB, free: 2.0 GiB)
23/07/21 13:12:29 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1433
23/07/21 13:12:29 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[3] at first at BaseCollectionExhaustJob.scala:440) (first 15 tasks are for partitions Vector(0))
23/07/21 13:12:29 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks resource profile 0
23/07/21 13:12:29 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0) (dp-services.internal.cloudapp.net, executor driver, partition 0, ANY, 6406 bytes) taskResourceAssignments Map()
23/07/21 13:12:29 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
23/07/21 13:12:29 INFO CodeGenerator: Code generated in 83.42606 ms
23/07/21 13:12:29 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1497 bytes result sent to driver
23/07/21 13:12:29 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 417 ms on dp-services.internal.cloudapp.net (executor driver) (1/1)
23/07/21 13:12:29 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
23/07/21 13:12:29 INFO DAGScheduler: ResultStage 0 (first at BaseCollectionExhaustJob.scala:440) finished in 0.624 s
23/07/21 13:12:29 INFO DAGScheduler: Job 0 is finished. Cancelling potential speculative or zombie tasks for this job
23/07/21 13:12:29 INFO TaskSchedulerImpl: Killing all running tasks in stage 0: Stage finished
23/07/21 13:12:29 INFO DAGScheduler: Job 0 finished: first at BaseCollectionExhaustJob.scala:440, took 0.656753 s
23/07/21 13:12:30 INFO CodeGenerator: Code generated in 17.713826 ms
23/07/21 13:12:30 INFO SparkUI: Stopped Spark web UI at http://54.255.154.146:4040
23/07/21 13:12:30 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
23/07/21 13:12:30 INFO MemoryStore: MemoryStore cleared
23/07/21 13:12:30 INFO BlockManager: BlockManager stopped
23/07/21 13:12:30 INFO BlockManagerMaster: BlockManagerMaster stopped
23/07/21 13:12:30 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
23/07/21 13:12:30 INFO SparkContext: Successfully stopped SparkContext
Exception in thread "main" java.lang.NoClassDefFoundError: scala/Product$class
at com.redislabs.provider.redis.RedisEndpoint.(RedisConfig.scala:28)at com.redislabs.provider.redis.RedisEndpoint.(RedisConfig.scala:36)
at org.apache.spark.sql.redis.RedisSourceRelation.(RedisSourceRelation.scala:36)
at org.apache.spark.sql.redis.DefaultSource.createRelation(DefaultSource.scala:42)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:353)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:325)
at org.apache.spark.sql.DataFrameReader.$anonfun$load$3(DataFrameReader.scala:307)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:307)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:225)
at org.sunbird.analytics.exhaust.BaseReportsJob.loadData(BaseReportsJob.scala:13)
at org.sunbird.analytics.exhaust.BaseReportsJob.loadData$(BaseReportsJob.scala:11)
at org.sunbird.analytics.exhaust.collection.ProgressExhaustJob$.loadData(ProgressExhaustJob.scala:19)
at org.sunbird.analytics.exhaust.collection.BaseCollectionExhaustJob.getUserCacheDF(BaseCollectionExhaustJob.scala:488)
at org.sunbird.analytics.exhaust.collection.BaseCollectionExhaustJob.getUserCacheDF$(BaseCollectionExhaustJob.scala:486)
at org.sunbird.analytics.exhaust.collection.ProgressExhaustJob$.getUserCacheDF(ProgressExhaustJob.scala:19)
at org.sunbird.analytics.exhaust.collection.BaseCollectionExhaustJob.$anonfun$execute$3(BaseCollectionExhaustJob.scala:108)
at org.ekstep.analytics.framework.util.CommonUtil$.time(CommonUtil.scala:538)
at org.sunbird.analytics.exhaust.collection.BaseCollectionExhaustJob.execute(BaseCollectionExhaustJob.scala:107)
at org.sunbird.analytics.exhaust.collection.BaseCollectionExhaustJob.execute$(BaseCollectionExhaustJob.scala:101)
at org.sunbird.analytics.exhaust.collection.ProgressExhaustJob$.execute(ProgressExhaustJob.scala:19)
at org.sunbird.analytics.exhaust.collection.BaseCollectionExhaustJob.$anonfun$main$1(BaseCollectionExhaustJob.scala:67)
at org.ekstep.analytics.framework.util.CommonUtil$.time(CommonUtil.scala:538)
at org.sunbird.analytics.exhaust.collection.BaseCollectionExhaustJob.main(BaseCollectionExhaustJob.scala:67)
at org.sunbird.analytics.exhaust.collection.BaseCollectionExhaustJob.main$(BaseCollectionExhaustJob.scala:57)
at org.sunbird.analytics.exhaust.collection.ProgressExhaustJob$.main(ProgressExhaustJob.scala:19)
at org.ekstep.analytics.job.JobExecutor$.main(JobExecutor.scala:16)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at optional.Application.callWithOptions(application.scala:251)
at optional.Application.callWithOptions$(application.scala:223)
at org.ekstep.analytics.job.JobExecutor$.callWithOptions(JobExecutor.scala:8)
at optional.Application.main(application.scala:258)
at optional.Application.main$(application.scala:255)
at org.ekstep.analytics.job.JobExecutor$.main(JobExecutor.scala:8)
at org.ekstep.analytics.job.JobExecutor.main(JobExecutor.scala)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:951)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1039)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1048)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: scala.Product$class
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:588)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
... 50 more
23/07/21 13:12:30 INFO ShutdownHookManager: Shutdown hook called
23/07/21 13:12:30 INFO ShutdownHookManager: Deleting directory /tmp/spark-d14eaa91-c037-49ae-be67-386bdc1c1fed
23/07/21 13:12:30 INFO ShutdownHookManager: Deleting directory /mount/data/analytics/tmp/spark-5c529328-8ecd-466a-b629-601115b748d0
23/07/21 13:12:30 INFO CassandraConnector: Disconnected from Cassandra cluster.
23/07/21 13:12:30 INFO SerialShutdownHooks: Successfully executed shutdown hook: Clearing session cache for C* connector
Job execution completed - progress-exhaust

[manjudr]

Are all the jobs failing? did you try running the other jobs? WFS or other report job (collection-summary-v2)?

[sravani-64]

Yes collection-summary-report also has errors

23/07/21 13:31:49 INFO SparkContext: Successfully stopped SparkContext
Exception in thread "main" java.lang.NoClassDefFoundError: scala/Product$class
at com.redislabs.provider.redis.RedisEndpoint.(RedisConfig.scala:28)
at com.redislabs.provider.redis.RedisEndpoint.(RedisConfig.scala:36)
at org.apache.spark.sql.redis.RedisSourceRelation.(RedisSourceRelation.scala:36)
at org.apache.spark.sql.redis.DefaultSource.createRelation(DefaultSource.scala:42)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:353)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:325)
at org.apache.spark.sql.DataFrameReader.$anonfun$load$3(DataFrameReader.scala:307)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:307)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:225)
at org.sunbird.analytics.job.report.BaseReportsJob.fetchData(BaseReportsJob.scala:97)
at org.sunbird.analytics.job.report.BaseReportsJob.fetchData$(BaseReportsJob.scala:95)
at org.sunbird.analytics.job.report.CollectionSummaryJobV2$.fetchData(CollectionSummaryJobV2.scala:22)
at org.sunbird.analytics.job.report.CollectionSummaryJobV2$.$anonfun$main$2(CollectionSummaryJobV2.scala:41)
at org.sunbird.analytics.job.report.CollectionSummaryJobV2$.getUserData(CollectionSummaryJobV2.scala:66)
at org.sunbird.analytics.job.report.CollectionSummaryJobV2$.prepareReport(CollectionSummaryJobV2.scala:109)
at org.sunbird.analytics.job.report.CollectionSummaryJobV2$.$anonfun$main$1(CollectionSummaryJobV2.scala:41)
at org.ekstep.analytics.framework.util.CommonUtil$.time(CommonUtil.scala:538)
at org.sunbird.analytics.job.report.CollectionSummaryJobV2$.main(CollectionSummaryJobV2.scala:41)
at org.ekstep.analytics.job.JobExecutor$.main(JobExecutor.scala:16)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at optional.Application.callWithOptions(application.scala:251)
at optional.Application.callWithOptions$(application.scala:223)
at org.ekstep.analytics.job.JobExecutor$.callWithOptions(JobExecutor.scala:8)
at optional.Application.main(application.scala:258)
at optional.Application.main$(application.scala:255)
at org.ekstep.analytics.job.JobExecutor$.main(JobExecutor.scala:8)
at org.ekstep.analytics.job.JobExecutor.main(JobExecutor.scala)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:951)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1039)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1048)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: scala.Product$class
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:588)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
... 43 more
23/07/21 13:31:49 INFO ShutdownHookManager: Shutdown hook called
23/07/21 13:31:49 INFO ShutdownHookManager: Deleting directory /mount/data/analytics/tmp/spark-6ce0c3c3-5d41-424e-bab0-4504b87cc3a0
23/07/21 13:31:49 INFO ShutdownHookManager: Deleting directory /tmp/spark-a74c26cb-2bf1-4fb8-8f4a-73c39be22a17
Job execution completed - collection-summary-report-v2

[manjudr]

It’s a Scala version mismatch issue. I’m not sure how you are running the job. Did you try running in a spark machine once? Is your Jenkins job pointed to the right spark version? Or Is your run job script pointed to right spark folder ? Did we make any Pom.xml file changes? Are artefacts built on the Java 11 version? Can we debug these things?

[sravani-64]

1. Yes, I executed the cronjob manually in the Spark machine using nohup command
2. We didn't trigger any cronjob from Jenkins.
3. We have only one spark version running, so the job points to same spark version
4. No changes were made in Pom.xml file
5. Yes, all the artifacts are built on Java version

[manjudr]

It's strange when everything appears fine, yet it doesn't work as expected. Have you thought about trying the latest version (4.10.5 +)?
Generally, this issue occurs if there is a Scala version conflict or if we pointed the jar to the wrong version of Spark or Java.

Could you kindly remove 'data-products', analytics-framework, and batch-models located at /mount/data/analytics/models-2.0/, from the spark then rebuild and deploy them in sequence?

[sravani-64]

I have removed the mentioned folders. Rebuit and deployed the sequence. Thank you
The jobs are being processed but failed with error com.microsoft.azure.storage.StorageException: The account being accessed does not support http.. Is there anyway to resolve this also?

[sravani-64]

@manjudr The above error is resolved by modifying the storage account configurations and progress-exhaust and response-exhaust jobs are successful.
But userinfo-exhaust is failed, giving an error as

java.util.NoSuchElementException: key [azureblob] not in the list of providers or apis: {}
at org.jclouds.ContextBuilder.newBuilder(ContextBuilder.java:175)
at org.sunbird.cloud.storage.service.AzureStorageService.(AzureStorageService.scala:11)
at org.sunbird.cloud.storage.factory.StorageServiceFactory$.getStorageService(StorageServiceFactory.scala:21)
at org.ekstep.analytics.framework.FrameworkContext.newStorageService(FrameworkContext.scala:54)
at org.ekstep.analytics.framework.FrameworkContext.getStorageService(FrameworkContext.scala:66)
at org.sunbird.analytics.exhaust.OnDemandExhaustJob.zipAndEncrypt(OnDemandExhaustJob.scala:175)
at org.sunbird.analytics.exhaust.OnDemandExhaustJob.$anonfun$processRequestEncryption$3(OnDemandExhaustJob.scala:150)
at scala.collection.immutable.List.map(List.scala:286)
at org.sunbird.analytics.exhaust.OnDemandExhaustJob.$anonfun$processRequestEncryption$1(OnDemandExhaustJob.scala:148)
at org.ekstep.analytics.framework.util.CommonUtil$.time(CommonUtil.scala:538)
at org.sunbird.analytics.exhaust.OnDemandExhaustJob.processRequestEncryption(OnDemandExhaustJob.scala:148)
at org.sunbird.analytics.exhaust.OnDemandExhaustJob.processRequestEncryption$(OnDemandExhaustJob.scala:147)
at org.sunbird.analytics.exhaust.collection.UserInfoExhaustJob$.processRequestEncryption(UserInfoExhaustJob.scala:8)
at org.sunbird.analytics.exhaust.OnDemandExhaustJob.saveRequest(OnDemandExhaustJob.scala:144)
at org.sunbird.analytics.exhaust.OnDemandExhaustJob.saveRequest$(OnDemandExhaustJob.scala:143)
at org.sunbird.analytics.exhaust.collection.UserInfoExhaustJob$.saveRequest(UserInfoExhaustJob.scala:8)
at org.sunbird.analytics.exhaust.OnDemandExhaustJob$$anon$1.$anonfun$get$1(OnDemandExhaustJob.scala:135)
at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23)
at org.ekstep.analytics.framework.util.CommonUtil$.time(CommonUtil.scala:538)
at org.sunbird.analytics.exhaust.OnDemandExhaustJob$$anon$1.get(OnDemandExhaustJob.scala:135)
at org.sunbird.analytics.exhaust.OnDemandExhaustJob$$anon$1.get(OnDemandExhaustJob.scala:133)
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1692)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177)
23/07/21 17:59:24 INFO ProducerConfig: ProducerConfig values:
acks = 1
batch.size = 100
bootstrap.servers = [10.50.10.4:9092]
buffer.memory = 33554432
client.id =
compression.type = snappy
connections.max.idle.ms = 540000
enable.idempotence = false
interceptor.classes = null
key.serializer = class org.apache.kafka.common.serialization.StringSerializer
linger.ms = 10
max.block.ms = 60000
max.in.flight.requests.per.connection = 5
max.request.size = 1048576
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
receive.buffer.bytes = 32768
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 60000
retries = 0
retry.backoff.ms = 100
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
send.buffer.bytes = 131072
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
transaction.timeout.ms = 60000
transactional.id = null
value.serializer = class org.apache.kafka.common.serialization.StringSerializer
23/07/21 17:59:24 INFO AppInfoParser: Kafka version : 1.0.1
23/07/21 17:59:24 INFO AppInfoParser: Kafka commitId : c0518aa65f25317e
23/07/21 17:59:24 INFO KafkaProducer: [Producer clientId=producer-1] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.
23/07/21 17:59:24 INFO SparkUI: Stopped Spark web UI at http://54.255.154.146:4040
23/07/21 17:59:24 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
23/07/21 17:59:24 INFO MemoryStore: MemoryStore cleared
23/07/21 17:59:24 INFO BlockManager: BlockManager stopped
23/07/21 17:59:24 INFO BlockManagerMaster: BlockManagerMaster stopped
23/07/21 17:59:24 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
23/07/21 17:59:24 INFO SparkContext: Successfully stopped SparkContext
23/07/21 17:59:24 INFO ShutdownHookManager: Shutdown hook called
23/07/21 17:59:24 INFO ShutdownHookManager: Deleting directory /tmp/spark-a35b9b41-8752-4104-b0e8-faf4294dcbbd
23/07/21 17:59:24 INFO ShutdownHookManager: Deleting directory /mount/data/analytics/tmp/spark-a4a2d593-7152-4912-9545-f28afb838f6f
23/07/21 17:59:24 INFO CassandraConnector: Disconnected from Cassandra cluster.
23/07/21 17:59:24 INFO SerialShutdownHooks: Successfully executed shutdown hook: Clearing session cache for C* connector
Job execution completed - userinfo-exhaust

Can You please help how to resolve the error in last row Zip, encrypt and upload failed

Thank you

[sravani-64]

Hi @manjudr , Could You please suggest how to resolve the issue related to userinfo-exhaust job?

[sravani-64]

userinfo-exhaust job is also running in the same Azure cloud

[manjudr]

What are the storage configurations of the user-info exhaust job?

[sravani-64]

Userinfo-exhaust reports are available in Azure cloud, storage account but the job is failure

[sravani-64]

@manjudr , we have setup reports on production environment running on 4.4 version but the jobs are failing with invalid request error

I have deleted data-products, analytics-framework, and batch-models and tried to build and deploy. But below errors are thrown during Jenkins build for AnalyticsCore, CoreDataProducts and EdDataProducts

But When I deploy with existing artifact, facing invalid request error and jobs are failed

[manjudr]

You mentioned earlier you are able to do the build and deployment right? https://github.com/orgs/Sunbird-Obsrv/discussions/87#discussioncomment-6508507

[sravani-64]

These are two different versions setup, sorry for the confusion. Could You Please help on userinfo-exhaust job issue and then later look into production build issue.
On Dev env, 4.10 version is setup on which progress, response exhaust jobs are running and userinfo job is throwing error
On Prod env, 4.4 version is in existence, on which we are trying to enable reports now and build is throwing error

[manjudr]

Is other exhaust jobs able to upload the report files to cloud storage?

[sravani-64]

Please let me know if I should check /mount/data/analytics/models-2.0 dev.conf file for userinfo configurations? If so, there is nothing related to userinfo-exhaust in dev.conf file. Other exhaust jobs report files are uploaded to cloud stoarge and the generated reports are also visible in the UI

[manjudr]

Kindly review the /mount/data/analytics/scripts/model-config.sh file. Within this file, you will find all the job configurations. Ensure you carefully compare and validate each of the configurations.

[sravani-64]

I have compared userinfo-exhaust configurations, but there are no differences

[manjudr]

@sravani-64 - If exhaust jobs are running fine, can we close these issues?

[sravani-64]

@manjudr userinfo exhaust job is failing with Redis exception errors
23/07/27 20:32:37 INFO DAGScheduler: Job 8 finished: count at BaseCollectionExhaustJob.scala:349, took 4.347716 s
org.apache.spark.sql.AnalysisException: cannot resolve 'consentprovideddate' given input columns: [batchName, batchid, block, cluster, collectionName, consentflag, courseid, district, email, orgname, p
hone, rootorgid, schoolname, schooludisecode, state, userid, username, usersubtype, usertype];
'Project [courseid#859, collectionName#984, batchid#899, batchName#1002, userid#858, username#60, state#29, district#30, orgname#31, email#1854, phone#1877, consentflag#1834, 'consentprovideddate, block#

[sravani-64]

@manjudr userinfo exhaust job is failing with Redis exception errors
23/07/27 20:32:37 INFO DAGScheduler: Job 8 finished: count at BaseCollectionExhaustJob.scala:349, took 4.347716 s
org.apache.spark.sql.AnalysisException: cannot resolve 'consentprovideddate' given input columns: [batchName, batchid, block, cluster, collectionName, consentflag, courseid, district, email, orgname, p
hone, rootorgid, schoolname, schooludisecode, state, userid, username, usersubtype, usertype];
'Project [courseid#859, collectionName#984, batchid#899, batchName#1002, userid#858, username#60, state#29, district#30, orgname#31, email#1854, phone#1877, consentflag#1834, 'consentprovideddate, block#

[manjudr]

Please verify whether the consentprovideddate property exists in Redis. If it is not defined, you should execute the user cache indexer ETL job to populate these properties.

[sravani-64]

We have executed the user cache indexer ETL job already but still the same errors are found in the userinfo job execution logs

[manjudr]

Has the job been executed successfully? Are there any errors in the Spark machine? Could you let me know if the data has been populated in the cache? How did we confirm whether the data was refreshed in Redis?

Is the consentprovideddate column present in the Cassandra user tables? And is the consentprovideddate property available in Redis for the users? Can you please debug all these?

[sravani-64]

@manjudr

1. Yes the ETL usercacheindexer job has been executed successfully
   
2. We have gathered logs from Spark machine in analytics@DP-Services-New4:/mount/data/analytics/content-snapshot/logs$ less 2023-07-31-shell-execution.log after running ETL usercacheindexer job
   
3. We have checked Keyspace from Redis (db12 is configured for reports) which has same number of key values before and after triggering the job
   
4. We have checked Cassandra db and found consentprovideddate is missing from two tables
   
   

[manjudr]

@sravani-64 - Please reach out or move the issue to learn team. They will be able to help you on resolving this issue. Since user management and all the report jobs are handled by learn team.

[sravani-64]

@manjudr we came to know that users from custodianorgID will not be able to generate userinfo-exhaust reports. Thanks for helping out. We can close the ticket