HIVE-24484: Upgrade Hadoop to 3.3.1

[belugabehr]

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

[ayushtkn]

Seems the tests are failing with
java.lang.NoSuchMethodError: org.jline.reader.impl.completer.StringsCompleter.<init>([Lorg/jline/reader/Candidate;)V
Should be fixable in the Hive Code itself?
If there is something required in the Hadoop Code, we can get that in now, 3.3.2 release is being planned out

[belugabehr]

@ayushtkn I'm not 100% sure what's going on there. I am working on upgrading jline as a separate task HIVE-25495 (#2617) and I'm hitting a similar issue there event though I thought I synchronized the versions between Hive and Hadoop, so I need to look into it more

[belugabehr]

OK, so the issue now is that Hadoop 3.3+ is using org.jline v3 and one of the Hive dependencies of sqline has a dependency on jline v2 which is causing a clash, well, not a clash per se, but it may be possible to address this by including both versions of jline since they have different namespaces. Ugh. I'll see if there is an updated sqline library or what that is doing exactly. Maybe it can be replaced with jline totally? I don't really know much about these libraries, just trying to rubik's cube this together.

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the dev@hive.apache.org list if the patch is in need of reviews.

[kgyrtkirk]

this PR is not making much progress - I think this in its current form will not work; or will not land soon:
I think it would make sesnse to consider:

• split this thing up into some pieces which we could get in...
• or even... upgrade to hadoop 3.1.MAX or 3.2.ANYTHING to grab some of the changes we have to cover in upgrading to 3.3.1
  instead of waiting this thing to get in with JDK11 support and everything? - what do you guys think?

[abstractdog]

what are the unresolved blockers of 3.3.1 upgrade at the moment?

[kgyrtkirk]

good question.... I don't know; but it seems like this PR have stalled!

this patch have:

• added a lib named jansi => this seems like an independent step
• removed the explicit guava usage from druid => could we upgrade guava separately? probably not...
• jline / jetty related things
• lots of other things

[ayushtkn]

or even... upgrade to hadoop 3.1.MAX or 3.2.ANYTHING to grab some of the changes we have to cover in upgrading to 3.3.1

Hadoop 3.1 line is EOL and 3.1.MAX & 3.2.Anything doesn't have Guava Shaded. So, guava version mismatch leads to even more issues,
From 3.3 only Guava & Protobuf are shaded in hadoop.

so the issue now is that Hadoop 3.3+ is using org.jline v3 and one of the Hive dependencies of sqline has a dependency on jline v2 which is causing a clash

If I decode @belugabehr's comment. The problem is only with Jline. If Jline v3 & v2 are compatible, and I see Jline is used by only Yarn-Client, So can we just not exclude the jline dependency while adding yarn-client?

[kgyrtkirk]

@ayushtkn: I've done some digging into the jline3 issue in #2617 (here) and I'm not sure if it was deliberate move to declare jline3 as a dependency of the hadoop-yarn-client artifact

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the dev@hive.apache.org list if the patch is in need of reviews.