Added HDFS functionality for streaming graphs from HDFS into Jasminegraph partitions. #252

muthumala19 · 2024-08-19T09:30:56Z

Added functionality to stream graph data from hdfs files and persist them in the Jasminegraph partitions.

…partitions with persisting them.

sonarqubecloud · 2024-08-19T09:33:59Z

Quality Gate failed

Failed conditions
14.2% Duplication on New Code (required ≤ 3%)

See analysis details on SonarCloud

miyurud · 2024-09-25T18:01:14Z

src/frontend/JasmineGraphFrontEnd.cpp

@@ -136,6 +141,10 @@ void *frontendservicesesion(void *dummyPt) {
    KafkaConnector *kstream;
    Partitioner graphPartitioner(numberOfPartitions, 1, spt::Algorithms::HASH);

+    //Initiate HDFS parameters
+    std::string hdfs_server_IP;


Suggested change

std::string hdfs_server_IP;

std::string hdfsServerIP;

Changed variable names, and method names into camel case notation.

miyurud · 2024-09-25T18:03:00Z

src/frontend/JasmineGraphFrontEnd.cpp

@@ -1251,6 +1267,137 @@ static void add_stream_kafka_command(int connFd, std::string &kafka_server_IP, c
    input_stream_handler_thread = thread(&StreamHandler::listen_to_kafka_topic, stream_handler);
 }

+void add_stream_hdfs_command(std::string masterIP,int connFd, std::string &hdfs_server_IP, std::thread &input_stream_handler_thread,


Suggested change

void add_stream_hdfs_command(std::string masterIP,int connFd, std::string &hdfs_server_IP, std::thread &input_stream_handler_thread,

void addStreamHdfsCommand(std::string masterIP,int connFd, std::string &hdfsServerIP, std::thread &inputStreamHandlerThread,

There is issue I see with function names and variable names. Please follow the approach I have used in my comments.

Changed variable names, and method names into camel case notation.

… methods into standard format.

miyurud · 2024-09-26T17:23:34Z

src/frontend/JasmineGraphFrontEnd.cpp

+        std::string file_path_s(file_path);
+        file_path_s = Utils::trim_copy(file_path_s);
+
+        frontend_logger.info("Reading HDFS configuration file: " + file_path_s);


Is this file part of JasmineGraph or is it something owned by the HDFS?

In this hdfs implementation, there are two types of HDFS servers.

Jasminegraph's Default HDFS server.

Custom HDFS servers from users.
users can use either one of those two types. if a user wants to refer to a custom hdfs server, then he needs to provide config file to that HDFS server. Here it refers to the custom hdfs server config file. currently this file should be put into the jasminegraph docker container and give the respective path.

miyurud · 2024-09-26T17:24:24Z

src/frontend/JasmineGraphFrontEnd.cpp

+
+        std::vector<std::string> vec = Utils::getFileContent(file_path_s);
+        for (const auto &item : vec) {
+            frontend_logger.info("Item: " + item);


I do not see a point of having log line here for the Item. Please remove it.

miyurud · 2024-09-26T17:26:03Z

src/frontend/JasmineGraphFrontEnd.cpp

+            if (item.length() > 0 && !(item.rfind("#", 0) == 0)) {
+                std::vector<std::string> vec2 = Utils::split(item, '=');
+                if (vec2.size() == 2) {
+                    if (vec2.at(0).compare("hdfs.host") == 0) {


This configuration file reading logic must be moved to some different class and different function. Do not clutter those details here.

miyurud · 2024-09-26T17:27:04Z

src/frontend/JasmineGraphFrontEnd.cpp

+    }
+
+    if (hdfs_server_IP.empty() || hdfs_port.empty()) {
+        frontend_logger.error("HDFS server IP or port not set correctly");


This error message does not have any meaning. You need to point out the erroneous value and the reason for why you say there is an error.l

Added more specific error handling with below code,

if (hdfsServerIp.empty()) { frontend_logger.error("HDFS server IP is not set or empty."); } if (hdfsPort.empty()) { frontend_logger.error("HDFS server port is not set or empty."); }

miyurud · 2024-09-26T17:27:29Z

src/frontend/JasmineGraphFrontEnd.cpp

+                        hdfs_port = vec2.at(1);
+                    }
+                } else {
+                    frontend_logger.error("Invalid line in configuration file: " + item);


Which configuration file are you referring to here?

In this hdfs implementation, there are two types of HDFS servers.

Jasminegraph's Default HDFS server.

Custom HDFS servers from users.
users can use either one of those two types. if a user wants to refer to a custom hdfs server, then he needs to provide config file to that HDFS server. Here it refers to the custom hdfs server config file.

miyurud · 2024-09-26T17:28:08Z

src/frontend/JasmineGraphFrontEnd.cpp

+
+    frontend_logger.info("HDFS server IP: " + hdfs_server_IP + " HDFS port: " + hdfs_port);
+
+    frontend_logger.info("Start serving HDFS command");


No meaning in this info log

miyurud · 2024-09-26T17:29:03Z

src/frontend/JasmineGraphFrontEnd.cpp

+    frontend_logger.info("HDFS server IP: " + hdfs_server_IP + " HDFS port: " + hdfs_port);
+
+    frontend_logger.info("Start serving HDFS command");
+    std::string message = "Send HDFS file path";


You should have short word commands than having this kind of lengthy sentences as the commands.

Shortened as "HDFS file path: "

miyurud · 2024-09-26T17:30:48Z

src/partitioner/local/MetisPartitioner.cpp

+    this->graphID = graphID;
+}
+
+void MetisPartitioner::loadFileChunks(string inputFilePath) {


This function name is misleading. It seems it does not do any meaningful processing.

This method is no longer there in the updated hdfs implementation.

miyurud · 2024-09-26T17:31:22Z

src/partitioner/local/MetisPartitioner.cpp

@@ -1009,3 +1009,125 @@ void MetisPartitioner::writeSerializedCompositeMasterFiles(std::string part) {
    masterFileMutex.unlock();
    partitioner_logger.log("Serializing done for central part " + part, "info");
 }
+
+void MetisPartitioner::setPathAndGraphID(int graphID) {


This function is not useful. I do not see any point of having such function.

This method is no longer there in the updated hdfs implementation.

…hreads for each partition.

…hreads for each partition: Added functionality to terminate unnecessary threads after streaming is done. and removed --network="host" part from the docker run command.

… Utils class, instead of using single array of DataPublisher objects.

… the partition.

… the workers, from master node.

miyurud · 2024-12-13T18:28:19Z

Please fix the commit messages in this PR following this blog

Addressed questioned parts in the PR[miyurud#252].

sonarqubecloud · 2024-12-21T06:41:33Z

Quality Gate failed

Failed conditions
7.7% Duplication on New Code (required ≤ 3%)
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

muthumala19 added 2 commits August 19, 2024 14:49

Implemented functionality to stream data from HDFS into jasminegraph …

7382d4f

…partitions with persisting them.

Added copyrights and license.

8042bb9

miyurud reviewed Sep 25, 2024

View reviewed changes

implemented hash partitioner class and make hdfs stream handler class…

4f4f8f9

… methods into standard format.

miyurud reviewed Sep 26, 2024

View reviewed changes

muthumala19 and others added 11 commits September 29, 2024 01:00

Implemented hash partitioner using separation of concern. using one t…

c61167e

…hreads for each partition.

Implemented hash partitioner using separation of concern. using one t…

1747a36

…hreads for each partition: Added functionality to terminate unnecessary threads after streaming is done. and removed --network="host" part from the docker run command.

Temporary commit

3ddb911

Temporary commit: implementation with data publisher objects

e56b68f

Merge branch 'miyurud:master' into master

77bdf05

Merge remote-tracking branch 'origin/master'

dd0cb6d

Created functionality to communicate between master and workers Using…

cb2fa97

… Utils class, instead of using single array of DataPublisher objects.

Added worker-side code to distribute data coming from file chunks, to…

1adc124

… the partition.

Modified functionality of fetching data from HDFS and sending them to…

e49d9f0

… the workers, from master node.

Added functionality to invalidate malformed paths to HDFS data sets.

73f2510

Added functionality to count edges correctly. (in HDFS implementation)

db4c398

Adjusted HDFS implementation for handling undirected graphs.

2ebdd1d

Addressed questioned parts in the PR[miyurud#252].

muthumala19 added a commit to muthumala19/jasminegraph that referenced this pull request Dec 19, 2024

Adjusted HDFS implementation for handling undirected graphs.

1a80e61

Addressed questioned parts in the PR[miyurud#252].

Added the copyright and license.

da21c26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added HDFS functionality for streaming graphs from HDFS into Jasminegraph partitions. #252

Added HDFS functionality for streaming graphs from HDFS into Jasminegraph partitions. #252

muthumala19 commented Aug 19, 2024

sonarqubecloud bot commented Aug 19, 2024

miyurud Sep 25, 2024

muthumala19 Dec 14, 2024

miyurud Sep 25, 2024

miyurud Sep 25, 2024

muthumala19 Dec 14, 2024

miyurud Sep 26, 2024

muthumala19 Dec 14, 2024

miyurud Sep 26, 2024

muthumala19 Dec 14, 2024

miyurud Sep 26, 2024

miyurud Sep 26, 2024

muthumala19 Dec 14, 2024

miyurud Sep 26, 2024

muthumala19 Dec 14, 2024

miyurud Sep 26, 2024

muthumala19 Dec 14, 2024

miyurud Sep 26, 2024

muthumala19 Dec 14, 2024

miyurud Sep 26, 2024

muthumala19 Dec 14, 2024

miyurud Sep 26, 2024

muthumala19 Dec 14, 2024

miyurud commented Dec 13, 2024

sonarqubecloud bot commented Dec 21, 2024

	void add_stream_hdfs_command(std::string masterIP,int connFd, std::string &hdfs_server_IP, std::thread &input_stream_handler_thread,
	void addStreamHdfsCommand(std::string masterIP,int connFd, std::string &hdfsServerIP, std::thread &inputStreamHandlerThread,


		frontend_logger.info("HDFS server IP: " + hdfs_server_IP + " HDFS port: " + hdfs_port);

		frontend_logger.info("Start serving HDFS command");

Added HDFS functionality for streaming graphs from HDFS into Jasminegraph partitions. #252

Are you sure you want to change the base?

Added HDFS functionality for streaming graphs from HDFS into Jasminegraph partitions. #252

Conversation

muthumala19 commented Aug 19, 2024

sonarqubecloud bot commented Aug 19, 2024

Quality Gate failed

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

miyurud commented Dec 13, 2024

sonarqubecloud bot commented Dec 21, 2024

Quality Gate failed