Sunbird 3.1 - Secor jobs are failing very frequently

[shishir84]

Hi Team,
We are using Sunbird 3.1 - data pipeline for our telemetry-related activities. For the last 6 months, we are facing below issues:

• secor jobs [ingest, raw, unique, derived] are going to the dead stage frequently. In random fashion. In this case, we are starting the secor jobs manually using the command line.

• Most of the time data is there in ingest s3 bucket but not in the raw s3 bucket and subsequent buckets [unique, derived] In this case, we are starting the secor jobs manually using the command line.

• Sometimes data is there in the raw s3 bucket but not in subsequent buckets [unique, derived] In this case, we are starting the secor jobs manually using the command line.

• Although data is there in the unique s3 bucket but not getting processed to the derived s3 bucket. In this case, using are running the derived time-spent shell script for missing dates.

• When we are running raw and unique secor jobs - it is skipping some dates.

Want to know how we can fix such issues permanently.

[shishir84]

this is a gentle reminder.

[manjudr]

@shishir84 - What error are you facing? Could you please post the error logs if possible? and which release of obsrv are you using it?

[shishir84]

Hi - we are using the sunbird 3.1 - DP module. We are not using obsrv. 2 weeks back we had a call with Rayalu and talked about the issues we facing. We will share the required either today eod or tomorrow.

[manjudr]

Sure. Please share the exception logs so it will be easy to debug or analyse the issue.

[anandp504]

@shishir84 I believe we closed out on these issues when we met. Please add the findings to this post and close this this issue.