Skip to content

Commit

Permalink
[Spark] [Test only] Fix Spark Master test in DeltaSourceSuite (#3088)
Browse files Browse the repository at this point in the history
#### Which Delta project/connector is this regarding?

- [X] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

Fix and re-enable a failing delta-spark test against Spark Master in
DeltaSourceSuite

## How was this patch tested?

Test only change.
  • Loading branch information
scottsand-db authored May 14, 2024
1 parent 9c88f8b commit b1b84d5
Showing 1 changed file with 6 additions and 3 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -1733,7 +1733,7 @@ class DeltaSourceSuite extends DeltaSourceSuiteBase
}
}

testSparkLatestOnly("startingVersion: user defined start works with mergeSchema") {
test("startingVersion: user defined start works with mergeSchema") {
withTempDir { inputDir =>
withTempView("startingVersionTest") {
spark.range(10)
Expand Down Expand Up @@ -1774,10 +1774,13 @@ class DeltaSourceSuite extends DeltaSourceSuiteBase
q.processAllAvailable()
checkAnswer(
sql("select * from startingVersionTest"),
((10 until 20).map(x => (x.toLong, x.toLong, None.toString)) ++
((10 until 20).map(x => (x.toLong, x.toLong, "null")) ++
(20 until 30).map(x => (x.toLong, x.toLong, x.toString)))
.toDF("id", "id2", "id3")
.selectExpr("id", "id2", "cast(id3 as long) as id3")
.selectExpr(
"id",
"id2",
"CASE WHEN id3 = 'null' THEN NULL ELSE cast(id3 as long) END as id3")
)
} finally {
q.stop()
Expand Down

0 comments on commit b1b84d5

Please sign in to comment.