Hadoop version conflict when supporting to use gazelle_plugin on Google Cloud Dataproc

[HongW2019]

Describe the bug
To support gazelle_plugin on Google Cloud Dataproc, we run spark-sql with gazelle_plugin v1.1.1-spark-3.1.1 enabled on
Google Cloud Dataproc 2.0 (CentOS 8, Hadoop 3.2.2, Spark 3.1.1), then met similar issues with EMR #368

[zhouyuan]

@HongW2019 i think the issue may be related with the hadoop version we used:
https://github.com/oap-project/gazelle_plugin/blob/master/pom.xml#L211

can you please try to build with 3.2.2?

[HongW2019]

@HongW2019 i think the issue may be related with the hadoop version we used:
https://github.com/oap-project/gazelle_plugin/blob/master/pom.xml#L211

can you please try to build with 3.2.2?

@zhouyuan Sure. After we built it with 3.2.2, met with the same exception with EMR.

[HongW2019]

@HongW2019 i think the issue may be related with the hadoop version we used:
https://github.com/oap-project/gazelle_plugin/blob/master/pom.xml#L211
can you please try to build with 3.2.2?

@zhouyuan Sure. After we built it with 3.2.2, met with the same exception with EMR.

@weiting-chen This result came from our re-built jar with 3.2.2, is this issue still related to Hadoop version conflict? wonder whether I need to re-produce it with your built jar with 3.2.2 if it is convenient.

[HongW2019]

Modify Hadoop version at properties then re-build jar would avoid Hadoop version conflict.
But if modify the version at profile then re-build with -Phadoop-3.2 then this jar will still bring exception.

[zhouyuan]

@HongW2019 can you please share the mvn info?
mvn -v should output the necessary info

[HongW2019]

mvn -v

sure @zhouyuan

(oapbuild) [root@bdpe-sky4 ~]# mvn -v
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-11T00:41:47+08:00)
Maven home: /opt/Beaver/maven
Java version: 1.8.0_112, vendor: Oracle Corporation
Java home: /opt/Beaver/jdk-8u112/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-1160.24.1.el7.x86_64", arch: "amd64", family: "unix"

[zhouyuan]

@HongW2019 seems quite old, could you please try to build the package with recent maven? I suspect this is due to the old maven which may not working well on reading properties from profile
here's the link for maven downloading
https://maven.apache.org/download.cgi

[HongW2019]

@HongW2019 seems quite old, could you please try to build the package with recent maven? I suspect this is due to the old maven which may not working well on reading properties from profile
here's the link for maven downloading
https://maven.apache.org/download.cgi

@zhouyuan After re-build jar with profile -Phadoop-3.2 with maven 3.6.3 instead of 3.3.9, run spark-shell with this jar found previous hadoop exception disappear.

[HongW2019]

After config Gazelle Plugin 1.2 on Dataproc 2.0 (CentOS 8, Hadoop 3.2.2, Spark 3.1.1) rightly, TPC-DS 99 queries have been successfully passed.

[HongW2019]

@zhouyuan @weiting-chen FYI. I think we can close this issue now.