hardcoded relative paths

[altaetran]

Hi, I noticed that the coordinator runner seems to have hardcoded locations of various jars relative to the HADOOP_HOME directory. This could work for some hadoop installations, but my installation has various jars located in different directories than the hardcoded relative paths used (i.e. $HADOOP_HOME/share/hadoop/yarn/ wouldn't work for me, but /hadoop/yarn/ would be the corresponding path for me). Would there be anyway to add an option for explicit references to the specific libraries required instead of having everything hardcoded relative to HADOOP_HOME? If this is not possible, would it be possible to describe all of the Hadoop dependencies so that I can sym link everything to a synthetic directory? Thanks!

Example error from trying to run:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream
at com.tencent.rss.coordinator.CoordinatorServer.initialization(CoordinatorServer.java:116)
at com.tencent.rss.coordinator.CoordinatorServer.(CoordinatorServer.java:53)
at com.tencent.rss.coordinator.CoordinatorServer.main(CoordinatorServer.java:68)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
... 3 more

[colinmjj]

@altaetran currently, the classpath for hadoop related is defined in start-coordinator.sh & start-shuffle-server.sh
I think there should be a guide to help user update the path with different environment.

[altaetran]

Hi Colin, Thanks for the response. I was able to change the paths explicitly to the corresponding absolute paths. I was able to get the start-coordinator.sh to run, but I am not ure if the shuffle server was able to run, as it ran normally, but unlike with the start-coordinator.sh, I was able to run the start-shuffle-server.sh multiple times without a warning about a process already being started. Moreover, upon trying to run in spark, normal map only tasks worked properly, but when I tried to do a shuffle task, the entire process started to hang without any progress (task completion bar did not even show up). Do you know if there is a way to verify if the start-shuffle-server.sh ran properly? Thanks!

[colinmjj]

@altaetran There has RestApi in both coordinator & shuffle server, you can check it to confirm if shuffle server do the heartbeat and available. The url should be http://coordinator:<rss.jetty.http.port>/metrics/server.
You can also check if shuffle server works as expected by RestApi.
I think document should be updated according to all problems from users, and I'll update it soon, maybe this week.

[jerqi]

Firestorm is now moved to ASF and renamed to Apache Uniffle (incubating).
This issue is solved by apache/uniffle#29

[altaetran]

Hi Colin, Thanks for the response. I was able to change the paths explicitly to the corresponding absolute paths. I was able to get the start-coordinator.sh to run, but I am not ure if the shuffle server was able to run, as it ran normally, but unlike with the start-coordinator.sh, I was able to run the start-shuffle-server.sh multiple times without a warning about a process already being started. Moreover, upon trying to run in spark, normal map only tasks worked properly, but when I tried to do a shuffle task, the entire process started to hang without any progress (task completion bar did not even show up). Do you know if there is a way to verify if the start-shuffle-server.sh ran properly? Thanks!

…

On Sun, May 22, 2022 at 10:05 PM Colin ***@***.***> wrote: @altaetran <https://github.com/altaetran> currently, the classpath for hadoop related is defined in start-coordinator.sh & start-shuffle-server.sh I think there should be a guide to help user update the path with different environment. — Reply to this email directly, view it on GitHub <#146 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABTQYBJKFA66SKXTSQMORFLVLLRWRANCNFSM5WSVIBOA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[jerryshao]

Hi @altaetran Firestorm project has already moved to Apache Uniffle (https://github.com/apache/incubator-uniffle), would you please create a new issue or discussion in the new repo? Thanks.