Spark 3.2 #587

echeipesh · 2022-06-30T05:02:36Z

Pushing for Spark 3.2

get tests green
update to spark-testing-base for more speed
get code review and/or cleanup the changes
figure out what to do with GeoTrellis layer data source
update python package
- ~~cleanup dependencies: disentangle the release wheel from pweave~~
- ~~grok the python environment, consider introducing poetry instead of conda~~
run some jobs in spark 3.2 environment

core/src/main/scala/org/locationtech/rasterframes/expressions/BinaryRasterFunction.scala

...rc/main/scala/org/locationtech/rasterframes/expressions/transformers/ReprojectGeometry.scala

pomadchin · 2022-07-10T20:26:12Z

I'm going to bump the min Spark version in GT as well locationtech/geotrellis#3389
Thogh it should not affect this PR much.

echeipesh · 2022-07-12T03:50:57Z

@pomadchin It'd be nice to get 3.6.3 out with spark 3.2.0 and then 3.7.0 with CES3 and spark 3.2.1. That'd allow this to sync with frameless 0.11 and later with 0.12+.

pomadchin · 2022-07-12T13:17:06Z

@echeipesh 🚀 locationtech/geotrellis#3471 + I'm dropping Spark 3.0 over there typelevel/frameless#637 (frameless will be published for 3.1.x, 3.2.x, and 3.3.x)

echeipesh · 2022-12-14T06:11:21Z

...est/scala/org/locationtech/rasterframes/datasource/geotrellis/GeoTrellisDataSourceSpec.scala

@@ -48,7 +48,8 @@ import org.scalatest.{BeforeAndAfterAll, Inspectors}

 import scala.math.{max, min}

-class GeoTrellisDataSourceSpec extends TestEnvironment with BeforeAndAfterAll with Inspectors with DataSourceOptions {
+trait GeoTrellisDataSourceSpec extends TestEnvironment with BeforeAndAfterAll with Inspectors with DataSourceOptions {


Very little motivation to fix this. Note that the push-down predicates also got busted with move to spark 3.2.

Can't imagine ever using this rather than reading COGs or even Zarr. Unless somebody would like to fix this I would like to suggest to remove this feature altogether to keep things tidy.

@metasim @pomadchin What do you guys think?

project/RFDependenciesPlugin.scala

Spark 3.2 Lets get it compiled, spark 2 support is well out the window anyway.

GT settings bring in jackson classes and with shading it was getting weird

this is a starting point

Also untangle the super weird inheritance relationship between the two

Made them more direct. Good for fixing things and better for performance because these versions don't need to create intermediate mask tiles.

This is a change but it's towards less surprising

- fixed weird init order in tests - all tests share same context now thanks to base - exclude scala-xml from tests

pomadchin

LGTM!

...rc/main/scala/org/locationtech/rasterframes/expressions/generators/RasterSourceToTiles.scala

pomadchin · 2023-01-03T14:13:49Z

...est/scala/org/locationtech/rasterframes/datasource/geotrellis/GeoTrellisDataSourceSpec.scala

@@ -48,7 +48,8 @@ import org.scalatest.{BeforeAndAfterAll, Inspectors}

 import scala.math.{max, min}

-class GeoTrellisDataSourceSpec extends TestEnvironment with BeforeAndAfterAll with Inspectors with DataSourceOptions {
+trait GeoTrellisDataSourceSpec extends TestEnvironment with BeforeAndAfterAll with Inspectors with DataSourceOptions {


core/src/main/scala/org/locationtech/rasterframes/expressions/package.scala

pomadchin · 2023-01-03T15:19:36Z

CI is dead? https://app.circleci.com/pipelines/github/locationtech/rasterframes?utm_campaign=workflow-failed&utm_medium=email&utm_source=notification

- Split out docs build into separate workflow. - Removed umlimit call (not needed)

core/src/main/scala/org/locationtech/rasterframes/expressions/package.scala

core/src/main/scala/org/locationtech/rasterframes/expressions/transformers/MaskExpression.scala

avoid reflection which is done at runtime by structural types

It needs more work at another time

echeipesh · 2023-01-19T21:47:16Z

I've been looking at the python side of the project enough to form strong enough opinions of it. I think it's actually going to need some work and it doesn't make sense to do it in this PR.

make the build PEP 517 compatible
ditch use of conda for poetry (we're not building a conda package here anyway)
separate the the pweave deps from the release deps
decide if shipping jars with .whl still makes sense
- it was a nice attempt at nicety but unless you start the process from ipython instead of pyspark it is just dead weight.
pyrasterframes python files are in the sbt project structure
- Kind of the only benefit of this is to couple the production of assembly jar with the building of the .whl
- Not sure the weirdness of this justifies the setup

So that's a lot of moving stuff around potentially and it would be much cleaner to make that it's own PR.

echeipesh commented Jun 30, 2022

View reviewed changes

core/src/main/scala/org/locationtech/rasterframes/expressions/BinaryRasterFunction.scala Outdated Show resolved Hide resolved

echeipesh commented Jun 30, 2022

View reviewed changes

...rc/main/scala/org/locationtech/rasterframes/expressions/transformers/ReprojectGeometry.scala Show resolved Hide resolved

DragonEnergy mentioned this pull request Nov 14, 2022

Need assembly.jar file and support for spark 3.0 Databricks 7.0+ #594

Open

echeipesh force-pushed the spark-3.2 branch 3 times, most recently from c264037 to bdbb98a Compare December 14, 2022 06:07

echeipesh marked this pull request as ready for review December 14, 2022 06:07

echeipesh commented Dec 14, 2022

View reviewed changes

project/RFDependenciesPlugin.scala Outdated Show resolved Hide resolved

metasim and others added 18 commits December 14, 2022 13:11

bumped dev version

30e7004

CI fix.

0b4309e

Dependency updates.

80e6992

Spark 3.1.3

d0e5bd5

bumped dev version

ff00b41

Spark 3.2 Lets get it compiled, spark 2 support is well out the window anyway.

withNewChildrenInternal

7f5e078

Try Aggregator implemtnation

160f351

more explicit

eb8ccb9

Fix UDF style Aggregates

a92ee4e

Bring in the Kryo setup

8da8bd7

GT settings bring in jackson classes and with shading it was getting weird

Register functions directly

ef2f4ee

this is a starting point

Bump versions

d12ace5

Landsat PDS is gone :(

aab5486

Fix Resample and ResampleNearest

a3ac4cf

Also untangle the super weird inheritance relationship between the two

fix masking functions

0214fa2

Made them more direct. Good for fixing things and better for performance because these versions don't need to create intermediate mask tiles.

Fix test: 6900 bit at position 4 is 1 -- expect NODATA after mask

285e03d

TileRasterizerAggregate expects column in rf_raster_proj order

725c9d5

This is a change but it's towards less surprising

Use spark-testing-base - core tests green

91468b4

- fixed weird init order in tests - all tests share same context now thanks to base - exclude scala-xml from tests

boop

43e8d3d

pomadchin approved these changes Jan 3, 2023

View reviewed changes

pomadchin force-pushed the spark-3.2 branch 2 times, most recently from d1ecf9b to a2d5a7a Compare January 3, 2023 15:14

pomadchin force-pushed the spark-3.2 branch from a2d5a7a to 2964d14 Compare January 3, 2023 15:21

Expressions constructors toSeq conversion

d1cfb99

pomadchin force-pushed the spark-3.2 branch from 2964d14 to d1cfb99 Compare January 3, 2023 15:22

echeipesh and others added 4 commits January 3, 2023 12:07

Downgrade scaffeine to 4.1.0 for JDK 8 support in caffeine 2.9

b28a10b

pyspark version 3.2.1

15b420c

why exclude log4j ? tests need it

05b4c44

GitHub actions build.

ec3c5f4

- Split out docs build into separate workflow. - Removed umlimit call (not needed)

echeipesh force-pushed the spark-3.2 branch from b551cb3 to ec3c5f4 Compare January 3, 2023 17:58

echeipesh and others added 2 commits January 3, 2023 13:01

Fix formatting

3a7b90f

Fix Expressions arity issue

4f24ad5

pomadchin force-pushed the spark-3.2 branch from aa4d871 to 4f24ad5 Compare January 3, 2023 18:20

Add .jvmopts

9be3cb6

pomadchin force-pushed the spark-3.2 branch from 286db2a to 9be3cb6 Compare January 3, 2023 19:08

metasim approved these changes Jan 3, 2023

View reviewed changes

Fix: Mask operations preserver the target tile cell type

61081b7

pomadchin approved these changes Jan 9, 2023

View reviewed changes

core/src/main/scala/org/locationtech/rasterframes/expressions/package.scala Outdated Show resolved Hide resolved

core/src/main/scala/org/locationtech/rasterframes/expressions/transformers/MaskExpression.scala Show resolved Hide resolved

echeipesh added 2 commits January 13, 2023 16:33

Pin GitHub Actions to ubuntu-20.04

b14adaa

Implement withNewChildrenInternal directly

df552b8

avoid reflection which is done at runtime by structural types

echeipesh force-pushed the spark-3.2 branch from 6798171 to e6cdf98 Compare January 19, 2023 21:37

Remove python build from CI

c5cf70c

It needs more work at another time

echeipesh force-pushed the spark-3.2 branch from e6cdf98 to c5cf70c Compare January 19, 2023 21:39

echeipesh merged commit 8331358 into develop Jan 19, 2023

echeipesh mentioned this pull request Jan 19, 2023

GitHub actions build. #564

Closed

echeipesh deleted the spark-3.2 branch February 24, 2023 22:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Spark 3.2 #587

Spark 3.2 #587

echeipesh commented Jun 30, 2022 •

edited

Loading

pomadchin commented Jul 10, 2022

echeipesh commented Jul 12, 2022

pomadchin commented Jul 12, 2022

echeipesh Dec 14, 2022

echeipesh Dec 14, 2022

pomadchin Jan 3, 2023

pomadchin left a comment

pomadchin Jan 3, 2023

pomadchin commented Jan 3, 2023

echeipesh commented Jan 19, 2023 •

edited

Loading

Spark 3.2 #587

Spark 3.2 #587

Conversation

echeipesh commented Jun 30, 2022 • edited Loading

pomadchin commented Jul 10, 2022

echeipesh commented Jul 12, 2022

pomadchin commented Jul 12, 2022

echeipesh Dec 14, 2022

Choose a reason for hiding this comment

echeipesh Dec 14, 2022

Choose a reason for hiding this comment

pomadchin Jan 3, 2023

Choose a reason for hiding this comment

pomadchin left a comment

Choose a reason for hiding this comment

pomadchin Jan 3, 2023

Choose a reason for hiding this comment

pomadchin commented Jan 3, 2023

echeipesh commented Jan 19, 2023 • edited Loading

echeipesh commented Jun 30, 2022 •

edited

Loading

echeipesh commented Jan 19, 2023 •

edited

Loading