Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Migrate download and ingest into job manager #3994

Merged
merged 3 commits into from
Mar 29, 2022
Merged

Migrate download and ingest into job manager #3994

merged 3 commits into from
Mar 29, 2022

Conversation

darionyaphet
Copy link
Contributor

@darionyaphet darionyaphet commented Mar 9, 2022 •
edited
Loading

What type of PR is this?

  • bug
  • feature
  • [*] enhancement

What problem(s) does this PR solve?

Issue(s) number:

#3733

Description:

SUBMIT COMMAND:

(root@nebula) [test]> FETCH PROP ON t 1 YIELD t.name
+--------+
| t.name |
+--------+
+--------+
Empty set (time spent 5474/6039 us)

Sun, 27 Mar 2022 07:39:34 CST

(root@nebula) [test]> SUBMIT JOB DOWNLOAD HDFS "hdfs://192.168.8.215:9000/tmp"
+------------+
| New Job Id |
+------------+
| 10         |
+------------+
Got 1 rows (time spent 2800/3514 us)

Sun, 27 Mar 2022 07:39:48 CST

(root@nebula) [test]> show jobs
+--------+------------+------------+----------------------------+----------------------------+
| Job Id | Command    | Status     | Start Time                 | Stop Time                  |
+--------+------------+------------+----------------------------+----------------------------+
| 10     | "DOWNLOAD" | "FINISHED" | 2022-03-26T23:39:48.000000 | 2022-03-26T23:39:52.000000 |
| 9      | "INGEST"   | "FINISHED" | 2022-03-26T16:42:44.000000 | 2022-03-26T16:42:44.000000 |
| 8      | "DOWNLOAD" | "FINISHED" | 2022-03-26T15:33:08.000000 | 2022-03-26T15:33:12.000000 |
| 7      | "INGEST"   | "FINISHED" | 2022-03-26T15:32:34.000000 | 2022-03-26T15:32:34.000000 |
| 6      | "INGEST"   | "FINISHED" | 2022-03-26T15:22:30.000000 | 2022-03-26T15:22:30.000000 |
| 5      | "DOWNLOAD" | "FINISHED" | 2022-03-26T15:22:16.000000 | 2022-03-26T15:22:21.000000 |
| 4      | "INGEST"   | "FINISHED" | 2022-03-26T15:13:46.000000 | 2022-03-26T15:13:46.000000 |
| 3      | "DOWNLOAD" | "FINISHED" | 2022-03-26T15:13:35.000000 | 2022-03-26T15:13:39.000000 |
+--------+------------+------------+----------------------------+----------------------------+
Got 8 rows (time spent 3269/4888 us)

Sun, 27 Mar 2022 07:39:54 CST

(root@nebula) [test]> SUBMIT JOB INGEST
+------------+
| New Job Id |
+------------+
| 11         |
+------------+
Got 1 rows (time spent 3055/3962 us)

Sun, 27 Mar 2022 07:40:04 CST

(root@nebula) [test]> show jobs
+--------+------------+------------+----------------------------+----------------------------+
| Job Id | Command    | Status     | Start Time                 | Stop Time                  |
+--------+------------+------------+----------------------------+----------------------------+
| 11     | "INGEST"   | "FINISHED" | 2022-03-26T23:40:04.000000 | 2022-03-26T23:40:04.000000 |
| 10     | "DOWNLOAD" | "FINISHED" | 2022-03-26T23:39:48.000000 | 2022-03-26T23:39:52.000000 |
| 9      | "INGEST"   | "FINISHED" | 2022-03-26T16:42:44.000000 | 2022-03-26T16:42:44.000000 |
| 8      | "DOWNLOAD" | "FINISHED" | 2022-03-26T15:33:08.000000 | 2022-03-26T15:33:12.000000 |
| 7      | "INGEST"   | "FINISHED" | 2022-03-26T15:32:34.000000 | 2022-03-26T15:32:34.000000 |
| 6      | "INGEST"   | "FINISHED" | 2022-03-26T15:22:30.000000 | 2022-03-26T15:22:30.000000 |
| 5      | "DOWNLOAD" | "FINISHED" | 2022-03-26T15:22:16.000000 | 2022-03-26T15:22:21.000000 |
| 4      | "INGEST"   | "FINISHED" | 2022-03-26T15:13:46.000000 | 2022-03-26T15:13:46.000000 |
| 3      | "DOWNLOAD" | "FINISHED" | 2022-03-26T15:13:35.000000 | 2022-03-26T15:13:39.000000 |
+--------+------------+------------+----------------------------+----------------------------+
Got 9 rows (time spent 2957/4441 us)

Sun, 27 Mar 2022 07:40:06 CST

(root@nebula) [test]> FETCH PROP ON t 1 YIELD t.name
+-----------+
| t.name    |
+-----------+
| "darion1" |
+-----------+
Got 1 rows (time spent 4353/5023 us)

Sun, 27 Mar 2022 07:40:09 CST



---------------------------------------------------------------------------------------------------------
DOWNLOAD AND INGEST COMMAND:

(root@nebula) [test]> FETCH PROP ON t 1 YIELD t.name
+--------+
| t.name |
+--------+
+--------+
Empty set (time spent 5332/6180 us)

Sun, 27 Mar 2022 07:41:39 CST

(root@nebula) [test]> DOWNLOAD HDFS "hdfs://192.168.8.215:9000/tmp"
+------------+
| New Job Id |
+------------+
| 12         |
+------------+
Got 1 rows (time spent 3028/3785 us)

Sun, 27 Mar 2022 07:41:59 CST

(root@nebula) [test]> show jobs
+--------+------------+------------+----------------------------+----------------------------+
| Job Id | Command    | Status     | Start Time                 | Stop Time                  |
+--------+------------+------------+----------------------------+----------------------------+
| 12     | "DOWNLOAD" | "FINISHED" | 2022-03-26T23:41:59.000000 | 2022-03-26T23:42:03.000000 |
| 11     | "INGEST"   | "FINISHED" | 2022-03-26T23:40:04.000000 | 2022-03-26T23:40:04.000000 |
| 10     | "DOWNLOAD" | "FINISHED" | 2022-03-26T23:39:48.000000 | 2022-03-26T23:39:52.000000 |
| 9      | "INGEST"   | "FINISHED" | 2022-03-26T16:42:44.000000 | 2022-03-26T16:42:44.000000 |
| 8      | "DOWNLOAD" | "FINISHED" | 2022-03-26T15:33:08.000000 | 2022-03-26T15:33:12.000000 |
| 7      | "INGEST"   | "FINISHED" | 2022-03-26T15:32:34.000000 | 2022-03-26T15:32:34.000000 |
| 6      | "INGEST"   | "FINISHED" | 2022-03-26T15:22:30.000000 | 2022-03-26T15:22:30.000000 |
| 5      | "DOWNLOAD" | "FINISHED" | 2022-03-26T15:22:16.000000 | 2022-03-26T15:22:21.000000 |
| 4      | "INGEST"   | "FINISHED" | 2022-03-26T15:13:46.000000 | 2022-03-26T15:13:46.000000 |
| 3      | "DOWNLOAD" | "FINISHED" | 2022-03-26T15:13:35.000000 | 2022-03-26T15:13:39.000000 |
+--------+------------+------------+----------------------------+----------------------------+
Got 10 rows (time spent 3408/4976 us)

Sun, 27 Mar 2022 07:42:17 CST

(root@nebula) [test]> INGEST
+------------+
| New Job Id |
+------------+
| 13         |
+------------+
Got 1 rows (time spent 2641/3205 us)

Sun, 27 Mar 2022 07:42:21 CST

(root@nebula) [test]> show jobs
+--------+------------+------------+----------------------------+----------------------------+
| Job Id | Command    | Status     | Start Time                 | Stop Time                  |
+--------+------------+------------+----------------------------+----------------------------+
| 13     | "INGEST"   | "FINISHED" | 2022-03-26T23:42:21.000000 | 2022-03-26T23:42:21.000000 |
| 12     | "DOWNLOAD" | "FINISHED" | 2022-03-26T23:41:59.000000 | 2022-03-26T23:42:03.000000 |
| 11     | "INGEST"   | "FINISHED" | 2022-03-26T23:40:04.000000 | 2022-03-26T23:40:04.000000 |
| 10     | "DOWNLOAD" | "FINISHED" | 2022-03-26T23:39:48.000000 | 2022-03-26T23:39:52.000000 |
| 9      | "INGEST"   | "FINISHED" | 2022-03-26T16:42:44.000000 | 2022-03-26T16:42:44.000000 |
| 8      | "DOWNLOAD" | "FINISHED" | 2022-03-26T15:33:08.000000 | 2022-03-26T15:33:12.000000 |
| 7      | "INGEST"   | "FINISHED" | 2022-03-26T15:32:34.000000 | 2022-03-26T15:32:34.000000 |
| 6      | "INGEST"   | "FINISHED" | 2022-03-26T15:22:30.000000 | 2022-03-26T15:22:30.000000 |
| 5      | "DOWNLOAD" | "FINISHED" | 2022-03-26T15:22:16.000000 | 2022-03-26T15:22:21.000000 |
| 4      | "INGEST"   | "FINISHED" | 2022-03-26T15:13:46.000000 | 2022-03-26T15:13:46.000000 |
| 3      | "DOWNLOAD" | "FINISHED" | 2022-03-26T15:13:35.000000 | 2022-03-26T15:13:39.000000 |
+--------+------------+------------+----------------------------+----------------------------+
Got 11 rows (time spent 2864/4388 us)

Sun, 27 Mar 2022 07:42:23 CST

(root@nebula) [test]> FETCH PROP ON t 1 YIELD t.name
+-----------+
| t.name    |
+-----------+
| "darion1" |
+-----------+
Got 1 rows (time spent 4644/5267 us)

Sun, 27 Mar 2022 07:42:25 CST

How do you solve it?

Special notes for your reviewer, ex. impact of this fix, design document, etc:

Checklist:

Tests:

  • Unit test(positive and negative cases)
  • Function test
  • Performance test
  • N/A

Affects:

  • Documentation affected (Please add the label if documentation needs to be modified.)
  • Incompatibility (If it breaks the compatibility, please describe it and add the label.）
  • If it's needed to cherry-pick (If cherry-pick to some branches is required, please label the destination version(s).)
  • Performance impacted: Consumes more CPU/Memory

Release notes:

Please confirm whether to be reflected in release notes and how to describe:

ex. Fixed the bug .....

@darionyaphet darionyaphet added the wip Solution: work in progress label Mar 9, 2022
@darionyaphet darionyaphet force-pushed the download-ingest-job-manager branch 5 times, most recently from b54e7ae to 02246d1 Compare March 16, 2022 06:57
@darionyaphet darionyaphet removed the wip Solution: work in progress label Mar 16, 2022
@darionyaphet darionyaphet force-pushed the download-ingest-job-manager branch 2 times, most recently from 0ff7e01 to d60e4cb Compare March 16, 2022 14:58
@Sophie-Xie Sophie-Xie requested a review from critical27 March 17, 2022 10:18
@Sophie-Xie Sophie-Xie added the ready-for-testing PR: ready for the CI test label Mar 17, 2022
@darionyaphet darionyaphet force-pushed the download-ingest-job-manager branch 5 times, most recently from d6f27c6 to a453978 Compare March 18, 2022 10:29
@darionyaphet darionyaphet changed the title migrate download and ingest into job manager Migrate download and ingest into job manager Mar 18, 2022
@darionyaphet darionyaphet force-pushed the download-ingest-job-manager branch from a453978 to 0dcead6 Compare March 18, 2022 10:40
@Sophie-Xie Sophie-Xie added this to the v3.1.0 milestone Mar 21, 2022
@panda-sheep
Copy link
Contributor

please fix conflicts

@darionyaphet darionyaphet force-pushed the download-ingest-job-manager branch 6 times, most recently from b3c8a70 to ecc85a8 Compare March 26, 2022 00:47
@darionyaphet darionyaphet requested a review from liuyu85cn March 26, 2022 01:35
@darionyaphet darionyaphet requested review from panda-sheep and a team March 26, 2022 01:35
@darionyaphet darionyaphet force-pushed the download-ingest-job-manager branch 4 times, most recently from b45967b to e66c144 Compare March 28, 2022 00:53
critical27
critical27 reviewed Mar 28, 2022
View reviewed changes
Copy link
Contributor

@critical27 critical27 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally LGTM, support ingest concurrently of different engines.

@HarrisChu
Copy link
Contributor

BTW, after this pr, no need configuration for ws_meta_http_port in graphd and ws_storage_port in metad
am I right?

@darionyaphet
Copy link
Contributor Author

BTW, after this pr, no need configuration for ws_meta_http_port in graphd and ws_storage_port in metad am I right?

no ReplaceHost still use it

@darionyaphet darionyaphet force-pushed the download-ingest-job-manager branch from d8026ef to b9aed06 Compare March 28, 2022 08:34
@HarrisChu
Copy link
Contributor

BTW, after this pr, no need configuration for ws_meta_http_port in graphd and ws_storage_port in metad am I right?

no ReplaceHost still use it

OK 👌🏻

@critical27
Copy link
Contributor

BTW, after this pr, no need configuration for ws_meta_http_port in graphd and ws_storage_port in metad am I right?

no ReplaceHost still use it

We don't need ws_meta_http_port/ws_storage_port? ReplaceHost is a pure http interface...

@darionyaphet
Copy link
Contributor Author

darionyaphet commented Mar 29, 2022 •
edited
Loading

BTW, after this pr, no need configuration for ws_meta_http_port in graphd and ws_storage_port in metad am I right?

no ReplaceHost still use it

We don't need ws_meta_http_port/ws_storage_port? ReplaceHost is a pure http interface...

replace method is register on http web service router

StorageHttpAdminHandler StorageHttpStatsHandler and StorageHttpPropertyHandler register on storage http web service

@darionyaphet darionyaphet force-pushed the download-ingest-job-manager branch 4 times, most recently from 955fdca to e51f9c3 Compare March 29, 2022 05:43
@CPWstatic CPWstatic merged commit 0649da8 into vesoft-inc:master Mar 29, 2022
@darionyaphet darionyaphet deleted the download-ingest-job-manager branch March 29, 2022 08:05
@jamieliu1023 jamieliu1023 mentioned this pull request Apr 2, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@critical27 critical27 critical27 left review comments

@panda-sheep panda-sheep panda-sheep approved these changes

@liuyu85cn liuyu85cn liuyu85cn approved these changes

@CPWstatic CPWstatic CPWstatic approved these changes

@dutor dutor Awaiting requested review from dutor dutor is a code owner

Assignees
No one assigned
Labels
ready for review ready-for-testing PR: ready for the CI test
Projects
None yet
Milestone
v3.1.0
Development

Successfully merging this pull request may close these issues.
7 participants
@darionyaphet @panda-sheep @HarrisChu @critical27 @CPWstatic @liuyu85cn @Sophie-Xie