[SPARK-32058][BUILD][SQL][test-hive1.2][test-hadoop2.7][FOLLOWUP] Set hadoop 2.7.4 for hive 1.2 profile #29006
Conversation
…r hive 1.2 profile
|
cc @dongjoon-hyun @cloud-fan @maropu thanks |
pom.xml
Outdated
| <datanucleus-core.version>3.2.10</datanucleus-core.version> | ||
| <hadoop.version>2.7.4</hadoop.version> | ||
| <curator.version>2.7.1</curator.version> | ||
| <commons-io.version>2.4</commons-io.version> |
There was a problem hiding this comment.
Does this mean the builds now pass even with -Phadoop-3.2 -Phive-1.2 but setting the Hadoop version to 2.7 implicitly?
There was a problem hiding this comment.
Ya. This looks completely wrong.
dev/run-tests-jenkins.py
Outdated
| # Switch the Hive profile based on the PR title: | ||
| if "test-hive1.2" in ghprb_pull_title: | ||
| os.environ["AMPLAB_JENKINS_BUILD_HIVE_PROFILE"] = "hive1.2" | ||
| os.environ["AMPLAB_JENKINS_BUILD_PROFILE"] = "hadoop2.7" |
There was a problem hiding this comment.
Let's don't do this but rather fail with an explicit error message. It will be confusing what Hive 1.2 uses its Hadoop version as.
There was a problem hiding this comment.
+1. thanks for the advice
|
I think you can give a shot to rather ban the profile combination with an explicit error message, for example, by using https://maven.apache.org/enforcer/enforcer-rules/bannedDependencies.html. |
dongjoon-hyun
left a comment
dongjoon-hyun
left a comment
There was a problem hiding this comment.
Hi, @yaooqinn .
Sorry, but I'm -1 for this PR. hive-1.2 is not a super profile. It should not override other profile.
|
One more thing, @yaooqinn ~ |
|
@dongjoon-hyun shall we remove the |
|
Nope, I don't think we should remove it now - it should be best to remove it when we completely remove |
|
If a PR wants to test hive 1.2, it must add both [test-hive1.2] and [test-hadoop2.7]? |
|
Yes, @cloud-fan ~ |
|
Test build #125016 has finished for PR 29006 at commit
|
yaooqinn
changed the title
|
retest this please |
|
Test build #125044 has started for PR 29006 at commit |
| <rules> | ||
| <bannedDependencies> | ||
| <excludes> | ||
| <exclude>org.apache.hadoop:*:[3.2.0,)</exclude> |
There was a problem hiding this comment.
Does this work in sbt run? It seems that PR builder is banned by this.
[info] Building Spark using SBT with these arguments: -Phadoop-2.7 -Phive-1.2 -Pspark-ganglia-lgpl -Pyarn -Pkubernetes -Phadoop-cloud -Phive-thriftserver -Phive -Pkinesis-asl -Pmesos test:package streaming-kinesis-asl-assembly/assembly
There was a problem hiding this comment.
It does not work for sbt runner, and I don't know how to enable this for it. Any advice？
There was a problem hiding this comment.
Let's just drop this PR then. We'll deprecate and remove Hive 1.2 profile in the near future anyway.
|
BTW, @yaooqinn . Please use a new JIRA ID because |
|
retest this please |
|
Test FAILed. |
6 participants
What changes were proposed in this pull request?
This PR set Hadoop 2.7.4 for hive 1.2 profile, because the default Hadoop 3.2.0 is incompatiable
Why are the changes needed?
The
test-hive1.2tag works wrongly these daysIt blocks #28912 (comment) and #28963 (comment)
Does this PR introduce any user-facing change?
No
How was this patch tested?
passing Jenkins with
test-hive-1.2tag