Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update spark330shim to use released lib #5848

Merged
merged 3 commits into from
Jun 16, 2022
Merged

Update spark330shim to use released lib #5848

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
2066a06
Revert "Temporarily allow 3.3.1 for 3.3.0 shims. (#5693)"
pxLi Jun 16, 2022
3ec8c37
update shim330 dep to released spark 330 libs
pxLi Jun 16, 2022
9382b0b
drop 330 from snapshot section in buildall
pxLi Jun 16, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions build/buildall
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -171,6 +171,7 @@ case $DIST_PROFILE in
313
320
321
330
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should drop it from snapshots?(WithDatabricks)) on L162

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch! updated

)
;;

Expand All @@ -179,6 +180,7 @@ case $DIST_PROFILE in
321cdh
312
320
330
)
;;

Expand Down
10 changes: 5 additions & 5 deletions dist/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,12 +46,12 @@
313,
320,
321,
321cdh
321cdh,
330
</noSnapshot.buildvers>
<snapshot.buildvers>
314,
322,
330
322
</snapshot.buildvers>
<databricks.buildvers>
312db,
Expand Down Expand Up @@ -91,14 +91,14 @@
- a downstream distro
- a snapshot
for testing while minimizing the build time
TODO add 3.3.0-SNAPSHOT
-->
<id>minimumFeatureVersionMix</id>
<properties>
<included_buildvers>
312,
320,
321cdh
321cdh,
330
</included_buildvers>
</properties>
</profile>
Expand Down
1 change: 1 addition & 0 deletions docs/additional-functionality/rapids-shuffle.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -286,6 +286,7 @@ In this section, we are using a docker container built using the sample dockerfi
| 3.2.0 | com.nvidia.spark.rapids.spark320.RapidsShuffleManager |
| 3.2.1 | com.nvidia.spark.rapids.spark321.RapidsShuffleManager |
| 3.2.1 CDH | com.nvidia.spark.rapids.spark321cdh.RapidsShuffleManager |
| 3.3.0 | com.nvidia.spark.rapids.spark330.RapidsShuffleManager |
| Databricks 9.1 | com.nvidia.spark.rapids.spark312db.RapidsShuffleManager |
| Databricks 10.4 | com.nvidia.spark.rapids.spark321db.RapidsShuffleManager |

Expand Down
2 changes: 1 addition & 1 deletion pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -863,7 +863,7 @@
<spark321cdh.version>3.2.1.3.2.7171000.0-3</spark321cdh.version>
<spark321db.version>3.2.1-databricks</spark321db.version>
<spark322.version>3.2.2-SNAPSHOT</spark322.version>
<spark330.version>3.3.1-SNAPSHOT</spark330.version>
<spark330.version>3.3.0</spark330.version>
<mockito.version>3.6.0</mockito.version>
<scala.plugin.version>4.3.0</scala.plugin.version>
<maven.jar.plugin.version>3.2.0</maven.jar.plugin.version>
Expand Down
4 changes: 1 addition & 3 deletions .../src/main/330/scala/com/nvidia/spark/rapids/shims/spark330/SparkShimServiceProvider.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,7 @@ import com.nvidia.spark.rapids.SparkShimVersion

object SparkShimServiceProvider {
val VERSION = SparkShimVersion(3, 3, 0)
// temporarily allow 3.3.1 while 3.3.0 release candidates are being produced
private val VERSION331 = SparkShimVersion(3, 3, 1)
val VERSIONNAMES = Seq(VERSION, VERSION331).flatMap(v => Seq(s"$v", s"$v-SNAPSHOT"))
val VERSIONNAMES = Seq(s"$VERSION", s"$VERSION-SNAPSHOT")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Snapshot support should have been removed here, posted a fix at #5858. I don't think it's must-fix for 22.06.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks!

}

class SparkShimServiceProvider extends com.nvidia.spark.rapids.SparkShimServiceProvider {
Expand Down
4 changes: 1 addition & 3 deletions ...plugin/src/test/330/scala/com/nvidia/spark/rapids/shims/spark330/Spark330ShimsSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,9 +24,7 @@ import org.apache.spark.sql.types.{DayTimeIntervalType, YearMonthIntervalType}

class Spark330ShimsSuite extends FunSuite {
test("spark shims version") {
// temporarily allow 3.3.1 while 3.3.0 release candidates are being produced
assert(SparkShimImpl.getSparkShimVersion === SparkShimVersion(3, 3, 0) ||
SparkShimImpl.getSparkShimVersion === SparkShimVersion(3, 3, 1))
assert(SparkShimImpl.getSparkShimVersion === SparkShimVersion(3, 3, 0))
}

test("shuffle manager class") {
Expand Down