[Question]: Why after running Spark task, there is no change of the database? (both on postgresql and mysql)

[huanghuihui0904]

Contact Details

huihuihuang0904@gmail.com

What would you like to ask or discuss?

After I ran the Spark task by the instructions you provide, there is no change of the database. Can you tell me how to solve that? Also I wonder how the program write the data to the database, in which way or in which java files?
As I have checked the error logs of my program, it just has the following two errors which I think not the key point.
In /opt/compass-v1.1.2/task-canal/logs

In /opt/compass-v1.1.2/task-portal/logs

Look forward to your reply!

[huanghuihui0904]

I want to add that now I fill all the table in mysql database(except blocklist), but the webpage still has no data.
I add the data by the following sqls:

INSERT INTO task_application VALUES (1, 'application_1712475671665_0002', 'Spark Pi', 'Spark Pi', 'Spark Pi', 'Spark', '2024-04-07 16:22:47', 1, 'http://192.168.106.192:18018/api/v1/applications/application_1712475671665_0001/1/logs', '2024-04-07 16:22:47', '2024-04-07 16:23:01');
INSERT INTO project VALUES (1, 'Spark Pi', 'description', 1, 1, '2024-04-07 16:22:47','2024-04-07 16:23:01');
INSERT INTO task VALUES (1,'Spark Pi',1,'Spark Pi',1,'Spark Pi','description',1,'SPARK',2,'2024-04-07 16:22:47', '2024-04-07 16:23:01');
INSERT INTO flow VALUES (1,'Spark Pi','description',1,1,1,'Spark Pi','2024-04-07 16:22:47', '2024-04-07 16:23:01');
INSERT INTO template VALUES(1,1,'cluster','advice','log','2024-04-07 16:23:01','2024-04-07 16:22:47')
INSERT INTO task_instance VALUES(1,'Spark Pi','Spark Pi','Spark Pi','2024-04-07 16:23:01','2024-04-07 16:23:01','2024-04-07 16:23:01.000000','FINISHED','SPARK',4,5,'worker group','schedule','2024-04-07 16:23:01','2024-04-07 16:23:01')
INSERT INTO task_datum VALUES(1,'Spark Pi','Spark Pi','Spark Pi','2024-04-07 16:23:01','baseline','2024-04-07 16:22:47')
INSERT INTO flink_task VALUES(1,'Spark Pi',1,'Spark Pi',1,'Spark Pi',1,0,'2024-04-07 16:22:47','2024-04-07 16:23:01')
INSERT INTO flink_task_app VALUES(1,'compass',1,'Spark Pi',1,'Spark Pi',1,'Spark Pi',1,'finish',1,'2024-04-07 16:23:01',1,'flink url',20,30,40,'Hadoop',100,'2024-04-07 16:22:47','2024-04-07 16:24:01',60,40,'queue','cluster',9,'compass','diagnosis',2,3,4,6,7,'Spark Pi','2024-04-07 16:22:47','2024-04-07 16:23:01')

Addtionally, I try the url http://192.****:7075/compass/api/v1/report/projects and the right response, which means there is no error in the mysql connection. But the table of database dolphinscheduler is empty, does this lead to the empty webpage?

[nilnon]

Contact Details

huihuihuang0904@gmail.com

What would you like to ask or discuss?

After I ran the Spark task by the instructions you provide, there is no change of the database. Can you tell me how to solve that? Also I wonder how the program write the data to the database, in which way or in which java files? As I have checked the error logs of my program, it just has the following two errors which I think not the key point. In /opt/compass-v1.1.2/task-canal/logs In /opt/compass-v1.1.2/task-portal/logs Look forward to your reply!

@huanghuihui0904 Check whether the canal dependency package has been downloaded?

[nilnon]

I want to add that now I fill all the table in mysql database(except blocklist), but the webpage still has no data. I add the data by the following sqls:

INSERT INTO task_application VALUES (1, 'application_1712475671665_0002', 'Spark Pi', 'Spark Pi', 'Spark Pi', 'Spark', '2024-04-07 16:22:47', 1, 'http://192.168.106.192:18018/api/v1/applications/application_1712475671665_0001/1/logs', '2024-04-07 16:22:47', '2024-04-07 16:23:01');
INSERT INTO project VALUES (1, 'Spark Pi', 'description', 1, 1, '2024-04-07 16:22:47','2024-04-07 16:23:01');
INSERT INTO task VALUES (1,'Spark Pi',1,'Spark Pi',1,'Spark Pi','description',1,'SPARK',2,'2024-04-07 16:22:47', '2024-04-07 16:23:01');
INSERT INTO flow VALUES (1,'Spark Pi','description',1,1,1,'Spark Pi','2024-04-07 16:22:47', '2024-04-07 16:23:01');
INSERT INTO template VALUES(1,1,'cluster','advice','log','2024-04-07 16:23:01','2024-04-07 16:22:47')
INSERT INTO task_instance VALUES(1,'Spark Pi','Spark Pi','Spark Pi','2024-04-07 16:23:01','2024-04-07 16:23:01','2024-04-07 16:23:01.000000','FINISHED','SPARK',4,5,'worker group','schedule','2024-04-07 16:23:01','2024-04-07 16:23:01')
INSERT INTO task_datum VALUES(1,'Spark Pi','Spark Pi','Spark Pi','2024-04-07 16:23:01','baseline','2024-04-07 16:22:47')
INSERT INTO flink_task VALUES(1,'Spark Pi',1,'Spark Pi',1,'Spark Pi',1,0,'2024-04-07 16:22:47','2024-04-07 16:23:01')
INSERT INTO flink_task_app VALUES(1,'compass',1,'Spark Pi',1,'Spark Pi',1,'Spark Pi',1,'finish',1,'2024-04-07 16:23:01',1,'flink url',20,30,40,'Hadoop',100,'2024-04-07 16:22:47','2024-04-07 16:24:01',60,40,'queue','cluster',9,'compass','diagnosis',2,3,4,6,7,'Spark Pi','2024-04-07 16:22:47','2024-04-07 16:23:01')

Addtionally, I try the url http://192.****:7075/compass/api/v1/report/projects and the right response, which means there is no error in the mysql connection. But the table of database dolphinscheduler is empty, does this lead to the empty webpage?

@huanghuihui0904 Part of the data of webpage comes from mysql, and part of it comes from elasticsearch/opensearch.

[nilnon]

@huanghuihui0904 You need to run all modules successfully and run spark tasks in dolphinscheduler so that task-portal can display data.

[huanghuihui0904]

But my opensearch can work well as I can open the web and see the data.
So do you mean the error with canal can lead to the problem: (1) after running spark task, no data written to mysql. (2)the webpage(actually the request http://192.*****:7075/compass/api/v1/report/statistics?projectName=) cannot get data.
Add some info to the request http://192.*****:7075/compass/api/v1/report/statistics?projectName=:
The responce as the following.

{
"code": 200,
"msg": "Succeed",
"data": null
}
And I find that this is the main request to get the data and then let the webpage show.

[huanghuihui0904]

@huanghuihui0904 You need to run all modules successfully and run spark tasks in dolphinscheduler so that task-portal can display data.

Do you mean the dolphinscheduler database? As I just fill the data in the compass database.

[nilnon]

@huanghuihui0904 You need to run all modules successfully and run spark tasks in dolphinscheduler so that task-portal can display data.

Do you mean the dolphinscheduler database? As I just fill the data in the compass database.

But my opensearch can work well as I can open the web and see the data. So do you mean the error with canal can lead to the problem: (1) after running spark task, no data written to mysql. (2)the webpage(actually the request http://192.*****:7075/compass/api/v1/report/statistics?projectName=) cannot get data. Add some info to the request http://192.*****:7075/compass/api/v1/report/statistics?projectName=: The responce as the following.

{
"code": 200,
"msg": "Succeed",
"data": null
}
And I find that this is the main request to get the data and then let the webpage show.

the /api/v1/report/statistics data mainly come from opensearch.