Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update spark-core, spark-mllib, spark-sql to 3.3.0 #637

Merged
merged 2 commits into from
Jul 12, 2022
Merged

Update spark-core, spark-mllib, spark-sql to 3.3.0 #637

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d094659
Update spark-core, spark-mllib, spark-sql to 3.3.0
typelevel-steward[bot] Jun 15, 2022
3fe486b
README.md update
pomadchin Jul 12, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,14 +29,14 @@ jobs:
strategy:
matrix:
os: [ubuntu-latest]
scala: [2.13.8, 2.12.15]
scala: [2.13.8, 2.12.16]
java: [temurin@8]
project: [root-spark30, root-spark31, root-spark32]
project: [root-spark31, root-spark32, root-spark33]
exclude:
- scala: 2.13.8
project: root-spark30
- scala: 2.13.8
project: root-spark31
- scala: 2.13.8
project: root-spark32
runs-on: ${{ matrix.os }}
steps:
- name: Checkout current branch (full)
Expand Down Expand Up @@ -104,7 +104,7 @@ jobs:
strategy:
matrix:
os: [ubuntu-latest]
scala: [2.12.15]
scala: [2.12.16]
java: [temurin@8]
runs-on: ${{ matrix.os }}
steps:
Expand Down Expand Up @@ -160,7 +160,7 @@ jobs:
strategy:
matrix:
os: [ubuntu-latest]
scala: [2.12.15]
scala: [2.12.16]
java: [temurin@8]
runs-on: ${{ matrix.os }}
steps:
Expand Down
3 changes: 2 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ The compatible versions of [Spark](http://spark.apache.org/) and
| 0.11.0* | 3.2.0 / 3.1.2 / 3.0.1| 2.x | 2.x | 2.12 / 2.13
| 0.11.1 | 3.2.0 / 3.1.2 / 3.0.1 | 2.x | 2.x | 2.12 / 2.13
| 0.12.0 | 3.2.1 / 3.1.3 / 3.0.3 | 2.x | 3.x | 2.12 / 2.13
| 0.13.0 | 3.3.0 / 3.2.1 /3.1.3 | 2.x | 3.x | 2.12 / 2.13

_\* 0.11.0 has broken Spark 3.1.2 and 3.0.1 artifacts published._

Expand All @@ -49,8 +50,8 @@ Starting 0.11 we introduced Spark cross published artifacts:
Artifact names examples:

* `frameless-dataset` (the latest Spark dependency)
* `frameless-dataset-spark32` (Spark 3.2.x dependency)
* `frameless-dataset-spark31` (Spark 3.1.x dependency)
* `frameless-dataset-spark30` (Spark 3.0.x dependency)

Versions 0.5.x and 0.6.x have identical features. The first is compatible with Spark 2.2.1 and the second with 2.3.0.

Expand Down
92 changes: 46 additions & 46 deletions build.sbt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
val sparkVersion = "3.2.1"
val sparkVersion = "3.3.0"
val spark32Version = "3.2.1"
val spark31Version = "3.1.3"
val spark30Version = "3.0.3"
val catsCoreVersion = "2.8.0"
val catsEffectVersion = "3.3.13"
val catsMtlVersion = "1.3.0"
Expand All @@ -11,7 +11,7 @@ val scalacheck = "1.16.0"
val scalacheckEffect = "1.0.4"
val refinedVersion = "0.10.1"

val Scala212 = "2.12.15"
val Scala212 = "2.12.16"
val Scala213 = "2.13.8"

ThisBuild / tlBaseVersion := "0.12"
Expand All @@ -23,23 +23,23 @@ ThisBuild / githubWorkflowArtifactUpload := false // doesn't work with scoverage

lazy val root = project.in(file("."))
.enablePlugins(NoPublishPlugin)
.aggregate(`root-spark32`, `root-spark31`, `root-spark30`, docs)
.aggregate(`root-spark33`, `root-spark32`, `root-spark31`, docs)

lazy val `root-spark33` = project
.in(file(".spark33"))
.enablePlugins(NoPublishPlugin)
.aggregate(core, cats, dataset, refined, ml)

lazy val `root-spark32` = project
.in(file(".spark32"))
.enablePlugins(NoPublishPlugin)
.aggregate(core, cats, dataset, refined, ml)
.aggregate(core, `cats-spark32`, `dataset-spark32`, `refined-spark32`, `ml-spark32`)

lazy val `root-spark31` = project
.in(file(".spark31"))
.enablePlugins(NoPublishPlugin)
.aggregate(core, `cats-spark31`, `dataset-spark31`, `refined-spark31`, `ml-spark31`)

lazy val `root-spark30` = project
.in(file(".spark30"))
.enablePlugins(NoPublishPlugin)
.aggregate(core, `cats-spark30`, `dataset-spark30`, `refined-spark30`, `ml-spark30`)

lazy val core = project
.settings(name := "frameless-core")
.settings(framelessSettings)
Expand All @@ -49,61 +49,61 @@ lazy val cats = project
.settings(catsSettings)
.dependsOn(dataset % "test->test;compile->compile;provided->provided")

lazy val `cats-spark32` = project
.settings(name := "frameless-cats-spark32")
.settings(sourceDirectory := (cats / sourceDirectory).value)
.settings(catsSettings)
.settings(spark32Settings)
.dependsOn(`dataset-spark32` % "test->test;compile->compile;provided->provided")

lazy val `cats-spark31` = project
.settings(name := "frameless-cats-spark31")
.settings(sourceDirectory := (cats / sourceDirectory).value)
.settings(catsSettings)
.settings(spark31Settings)
.dependsOn(`dataset-spark31` % "test->test;compile->compile;provided->provided")

lazy val `cats-spark30` = project
.settings(name := "frameless-cats-spark30")
.settings(sourceDirectory := (cats / sourceDirectory).value)
.settings(catsSettings)
.settings(spark30Settings)
.dependsOn(`dataset-spark30` % "test->test;compile->compile;provided->provided")

lazy val dataset = project
.settings(name := "frameless-dataset")
.settings(datasetSettings)
.settings(sparkDependencies(sparkVersion))
.dependsOn(core % "test->test;compile->compile")

lazy val `dataset-spark31` = project
.settings(name := "frameless-dataset-spark31")
lazy val `dataset-spark32` = project
.settings(name := "frameless-dataset-spark32")
.settings(sourceDirectory := (dataset / sourceDirectory).value)
.settings(datasetSettings)
.settings(sparkDependencies(spark31Version))
.settings(spark31Settings)
.settings(sparkDependencies(spark32Version))
.settings(spark32Settings)
.dependsOn(core % "test->test;compile->compile")

lazy val `dataset-spark30` = project
.settings(name := "frameless-dataset-spark30")
lazy val `dataset-spark31` = project
.settings(name := "frameless-dataset-spark31")
.settings(sourceDirectory := (dataset / sourceDirectory).value)
.settings(datasetSettings)
.settings(sparkDependencies(spark30Version))
.settings(spark30Settings)
.settings(sparkDependencies(spark31Version))
.settings(spark31Settings)
.dependsOn(core % "test->test;compile->compile")

lazy val refined = project
.settings(name := "frameless-refined")
.settings(refinedSettings)
.dependsOn(dataset % "test->test;compile->compile;provided->provided")

lazy val `refined-spark32` = project
.settings(name := "frameless-refined-spark32")
.settings(sourceDirectory := (refined / sourceDirectory).value)
.settings(refinedSettings)
.settings(spark32Settings)
.dependsOn(`dataset-spark32` % "test->test;compile->compile;provided->provided")

lazy val `refined-spark31` = project
.settings(name := "frameless-refined-spark31")
.settings(sourceDirectory := (refined / sourceDirectory).value)
.settings(refinedSettings)
.settings(spark31Settings)
.dependsOn(`dataset-spark31` % "test->test;compile->compile;provided->provided")

lazy val `refined-spark30` = project
.settings(name := "frameless-refined-spark30")
.settings(sourceDirectory := (refined / sourceDirectory).value)
.settings(refinedSettings)
.settings(spark30Settings)
.dependsOn(`dataset-spark30` % "test->test;compile->compile;provided->provided")

lazy val ml = project
.settings(name := "frameless-ml")
.settings(mlSettings)
Expand All @@ -113,26 +113,26 @@ lazy val ml = project
dataset % "test->test;compile->compile;provided->provided"
)

lazy val `ml-spark31` = project
.settings(name := "frameless-ml-spark31")
lazy val `ml-spark32` = project
.settings(name := "frameless-ml-spark32")
.settings(sourceDirectory := (ml / sourceDirectory).value)
.settings(mlSettings)
.settings(sparkMlDependencies(spark31Version))
.settings(spark31Settings)
.settings(sparkMlDependencies(spark32Version))
.settings(spark32Settings)
.dependsOn(
core % "test->test;compile->compile",
`dataset-spark31` % "test->test;compile->compile;provided->provided"
`dataset-spark32` % "test->test;compile->compile;provided->provided"
)

lazy val `ml-spark30` = project
.settings(name := "frameless-ml-spark30")
lazy val `ml-spark31` = project
.settings(name := "frameless-ml-spark31")
.settings(sourceDirectory := (ml / sourceDirectory).value)
.settings(mlSettings)
.settings(sparkMlDependencies(spark30Version))
.settings(spark30Settings)
.settings(sparkMlDependencies(spark31Version))
.settings(spark31Settings)
.dependsOn(
core % "test->test;compile->compile",
`dataset-spark30` % "test->test;compile->compile;provided->provided"
`dataset-spark31` % "test->test;compile->compile;provided->provided"
)

lazy val docs = project
Expand Down Expand Up @@ -258,12 +258,12 @@ lazy val framelessSettings = Seq(
},
) ++ consoleSettings

lazy val spark30Settings = Seq(
lazy val spark31Settings = Seq(
crossScalaVersions := Seq(Scala212)
)

lazy val spark31Settings = Seq(
crossScalaVersions := Seq(Scala212)
lazy val spark32Settings = Seq(
mimaPreviousArtifacts := Set.empty
)
Comment on lines +265 to 267
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a rule-of-thumb, you should never set mimaPreviousArtifacts := Set.empty, this is easy to forget and break binary-compatibiliy in a future release by mistake.

There are basically two situations:

  1. You are not publishing the module, so use .enablePlugins(NoPublishPlugin) to disable MiMa.
  2. It is a new module that you have not published yet. So use this setting to indicate which version it will be introduced, and MiMa will automatically start from that version. 
     tlVersionIntroduced := Map("2.12" -> "0.13.0", "2.13" -> "0.13.0")

Copy link
Member

@pomadchin pomadchin Jul 17, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@armanbilge thanks! Is this intel documented anywhere?

What will be the setting for the artifact we publish for the first time and it has not been published before?
To be clear: 3.2 is the new artifact and we don't need to compare it with the previous version.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this intel documented anywhere?

Sort of. This particular setting is documented here:
https://typelevel.org/sbt-typelevel/faq.html#how-do-i-indicate-the-first-version-that-i-published-scala-3-artifacts-for

Although perhaps it can use some other examples like for your situation.

What will be the setting for the artifact we publish (first time) and it has not been published before?

What is the first version you plan to publish it in?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@armanbilge o i c, 0.13 is the first version, so we need to add tlVersionIntroduced := Map("2.12" -> "0.13.0", "2.13" -> "0.13.0") as an indication that the first artifact was published as the 0.13 version for scala 2.12 and 2.13.

Makes sense now, thanks!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, exactly, you got it! :) And this way you don't need to remember to change it again after publishing 0.13.0, MiMa will start working automatically.


lazy val consoleSettings = Seq(
Expand Down Expand Up @@ -331,7 +331,7 @@ ThisBuild / githubWorkflowBuildPreamble ++= Seq(
)
)

val roots = List("root-spark30", "root-spark31", "root-spark32")
val roots = List("root-spark31", "root-spark32", "root-spark33")
ThisBuild / githubWorkflowBuildMatrixAdditions +=
"project" -> roots
ThisBuild / githubWorkflowArtifactDownloadExtraKeys += "project"
Expand Down