[KYUUBI #298]Add Support for Apache Spark 3.1.1

[yaooqinn]

❨?❩

Why are the changes needed?

Spark 3.1.1 released, it looks more stable than the first minor release of Spark 3.x, and a lot of great features were involved.

In this PR, I add a profile for spark-3.1.1 support, and temporary we only support hadoop2.7

TODOS:

• Add Hadoop 3.2 support with some dependency issues
• Enable Iceberg tests for spark-3.1.1 after Apache iceberg‘s next release
• Using spark-3.1.1 as default
• Restore HiveResult string?

fix #298

How was this patch tested?

• Add some test cases that check the changes thoroughly including negative and positive cases if possible

• Add screenshots for manual tests if appropriate

• Run test locally before make a pull request

[github-actions]

Scan Summary

Tool	Critical	High	Medium	Low	Status
Security Audit for Infrastructure	0	0	0	0	✅
Java Source Analyzer	0	0	0	1	✅
Class File Analyzer	0	6	0	0	❌
Python Security Analysis	3	0	0	0	❌
Python Source Analyzer	0	0	0	0	✅
Scala Security Audit	0	6	0	0	❌
Shell Script Analysis	0	0	0	0	✅

Recommendation

Please review the findings from Code scanning alerts before approving this pull request. You can also configure the build rules or add suppressions to customize this bot 👍

[github-actions]

Scan Summary

Tool	Critical	High	Medium	Low	Status
Dependency Scan (java)	0	2	5	0	✅
Security Audit for Infrastructure	0	0	0	0	✅
Java Source Analyzer	0	0	0	1	✅
Class File Analyzer	0	6	0	0	❌
Python Security Analysis	3	0	0	0	❌
Python Source Analyzer	0	0	0	0	✅
Scala Security Audit	0	6	0	0	❌
Shell Script Analysis	0	0	0	0	✅

Recommendation

Please review the findings from Code scanning alerts before approving this pull request. You can also configure the build rules or add suppressions to customize this bot 👍

[github-actions]

Scan Summary

Tool	Critical	High	Medium	Low	Status
Dependency Scan (java)	0	2	4	0	✅
Security Audit for Infrastructure	0	0	0	0	✅
Java Source Analyzer	0	0	0	1	✅
Class File Analyzer	0	6	0	0	❌
Python Security Analysis	3	0	0	0	❌
Python Source Analyzer	0	0	0	0	✅
Scala Security Audit	0	6	0	0	❌
Shell Script Analysis	0	0	0	0	✅

Recommendation

Please review the findings from Code scanning alerts before approving this pull request. You can also configure the build rules or add suppressions to customize this bot 👍

[codecov]

Codecov Report

Merging #366 (7b332ec) into master (aa878e4) will increase coverage by 0.19%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #366      +/-   ##
==========================================
+ Coverage   80.49%   80.69%   +0.19%     
==========================================
  Files         101       99       -2     
  Lines        3640     3620      -20     
  Branches      428      426       -2     
==========================================
- Hits         2930     2921       -9     
+ Misses        492      479      -13     
- Partials      218      220       +2     

Impacted Files	Coverage Δ
...rg/apache/kyuubi/engine/spark/SparkSQLEngine.scala	64.81% <100.00%> (-5.00%)	⬇️
...org/apache/kyuubi/ha/client/ServiceDiscovery.scala	61.36% <0.00%> (-3.68%)	⬇️
...ache/kyuubi/ha/client/KyuubiServiceDiscovery.scala
...ache/kyuubi/ha/client/EngineServiceDiscovery.scala
...org/apache/kyuubi/operation/ExecuteStatement.scala	70.27% <0.00%> (+2.70%)	⬆️

[yaooqinn]

cc @wangyum @ulysses-you @pan3793 @Honglun @zwangsheng @xjzrc thanks for reviewing

[pan3793]

LTGM

[yaooqinn]

thanks all, merged to master for v1.1.0