How to automate the RunReportJob programmatically

[rakeshSgr]

Hi,

As part of the 6.0 release, we are automating the creation of admin reports and program dashboard reports using a Python script. In the script, we make API calls to create backend and frontend configurations.

After running the script, in order to make the reports available, we need to execute the RunReportJob service in Jenkins. This service, accessible at http://10.20.0.14:8080/jenkins/job/Deploy/job/dev/job/DataPipeline/job/RunReportJob/, is responsible for executing the Druid query associated with the reports. It retrieves the query results and converts them into CSV and JSON formats. These output files are then pushed to the cloud storage.

Running the RunReportJob service ensures that the latest data is fetched and made available for generating the reports. Currently, we need to manually run this job for each report.

@manjudr @kumarks1122 @santhosh-tg @tsprasath
When we create a new report, we need to manually run it via Jenkins to create a cron entry.
Please let us know, how we do it if not via Jenkins

CC : @aks30 @vijiurs @adithyadinesh0412

[manjudr]

@rakeshSgr

There are multiple ways to automate it:

1. You can choose to modify your Jenkins jobs scheduler section to run the job once a day, according to your preferred frequency. This configuration will ensure that the Jenkins job is triggered on a daily basis. As an example, you can refer to the ETL Jobs and the Content Cache Updater Jenkins jobs.

2. Another approach is to create a simple Ansible role that configures the cron job within the machine. This job should only be deployed during the provisioning or deployment time.

[rakeshSgr]

@manjudr Could you please let us know a convenient time for us to connect? It would be beneficial as we can discuss the problem we are facing and work towards finding a solution together.

[rakeshSgr]

@manjudr @anandp504
In release 6.0, we are faced with the task of creating over 60 charts ( around 10 reports). Currently, we have to run the RunReportJob 60 times individually to generate the necessary data. This process is time-consuming and inefficient.

To verify, We have created the report called "Test Report for Auto Update" in staging on July 7th, till date report data has not been generated.

We kindly request your assistance in resolving this issue. We are seeking a more streamlined and efficient solution to create all the required reports in a more automated and consolidated manner.

@aks30 @vijiurs @adithyadinesh0412

[rakeshSgr]

@santhosh-tg and @tsprasath Please enable the report job in staging, so that we confirm its working or not.
cc : @manjudr @aks30

[tsprasath]

@rakeshSgr http://10.20.0.14:8080/jenkins/job/Deploy/job/staging/job/DataPipeline/job/RunReportjob/

[rakeshSgr]

@manjudr is the same job you are referring to?
@tsprasath is this job run every day?

[manjudr]

Yes @rakeshSgr

[rakeshSgr]

@tsprasath is the job run every day?