[BUILD] Revise Kyuubi Spark engine shaded #3776
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pan3793
changed the title
Member
Author
|
There are incompatible changes in Netty for early Spark release, then the shade of Netty is required. |
github-actions
bot
added
kind:infra
pan3793
force-pushed
the
spark-engine-shade
branch
from
03302c0 to
19382ef
Compare
Codecov Report
@@ Coverage Diff @@
## master #3776 +/- ##
============================================
- Coverage 52.87% 52.85% -0.02%
Complexity 13 13
============================================
Files 496 496
Lines 27971 27967 -4
Branches 3857 3856 -1
============================================
- Hits 14789 14782 -7
Misses 11788 11788
- Partials 1394 1397 +3
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
Member
Author
|
ping @hddong, it's ready for reviewing and testing |
hddong
reviewed
Nov 10, 2022
| Copyright 2006-2010 The Apache Software Foundation. | ||
|
|
||
| Google Guava Version 18.0 | ||
| * Copyright (C) 2009 The Guava Authors |
Contributor
There was a problem hiding this comment.
Accidentally deleted? Seems Guava still be include.
Member
Author
There was a problem hiding this comment.
The recent version Guava does not ship NOTICE in the jar
Contributor
There was a problem hiding this comment.
Get, LGTM, I will have a test locally.
hddong
approved these changes
Nov 11, 2022
Contributor
|
Checked locally, it works as expected. |
Member
Author
|
Thanks, merging to master |
3 tasks
pan3793
added a commit
that referenced
this pull request
Dec 2, 2022
This PR revises the shaded rule of spark engine module, especially to make sure that netty native libs is shaded properly.
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
```
(kyuubi) ➜ apache-kyuubi git:(spark-engine-shade) tree externals/kyuubi-spark-sql-engine/target/unpacked -L 5 | grep -v class
externals/kyuubi-spark-sql-engine/target/unpacked
├── META-INF
│ ├── LICENSE
│ ├── MANIFEST.MF
│ ├── NOTICE
│ ├── io.netty.versions.properties
│ ├── native
│ │ ├── liborg_apache_kyuubi_shade_netty_transport_native_epoll_aarch_64.so
│ │ └── liborg_apache_kyuubi_shade_netty_transport_native_epoll_x86_64.so
│ ├── services
│ │ ├── org.apache.kyuubi.shade.io.grpc.LoadBalancerProvider
│ │ ├── org.apache.kyuubi.shade.io.grpc.ManagedChannelProvider
│ │ ├── org.apache.kyuubi.shade.io.grpc.NameResolverProvider
│ │ ├── org.apache.kyuubi.shade.io.grpc.ServerProvider
│ │ ├── org.apache.kyuubi.shade.io.vertx.core.spi.launcher.CommandFactory
│ │ ├── org.apache.spark.status.AppHistoryServerPlugin
│ │ └── reactor.blockhound.integration.BlockHoundIntegration
│ ├── versions
│ │ └── 11
│ │ └── io
│ │ └── vertx
│ └── vertx
│ └── vertx-version.txt
├── kyuubi-version-info.properties
├── log4j2-defaults.xml
├── org
│ └── apache
│ ├── kyuubi
│ │ ├── cli
│ │ ├── config
│ │ │ └── internal
│ │ ├── engine
│ │ │ └── spark
│ │ ├── events
│ │ │ └── handler
│ │ ├── ha
│ │ │ └── client
│ │ ├── operation
│ │ │ ├── log
│ │ │ └── meta
│ │ ├── reflection
│ │ ├── service
│ │ │ └── authentication
│ │ ├── session
│ │ ├── shade
│ │ │ ├── android
│ │ │ ├── com
│ │ │ ├── io
│ │ │ ├── net
│ │ │ └── org
│ │ └── util
│ └── spark
│ ├── api
│ │ └── python
│ ├── kyuubi
│ └── ui
└── python
├── execute_python.py
└── kyuubi_util.py
40 directories, 211 files
```
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes #3776 from pan3793/spark-engine-shade.
Closes #3776
6a6e2a6 [Cheng Pan] nit
e247923 [Cheng Pan] 1
a53b7c0 [Cheng Pan] nit
19382ef [Cheng Pan] [BUILD] Revisit Kyuubi Spark engine shaded
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
Member
Author
|
Cherry picked to branch-1.6 as user reported that jetcd class conflict w/ TiSpark |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are the changes needed?
This PR revises the shaded rule of spark engine module, especially to make sure that netty native libs is shaded properly.
How was this patch tested?
Add some test cases that check the changes thoroughly including negative and positive cases if possible
Add screenshots for manual tests if appropriate