Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Need support on Spark 3.2.x, Spark 3.3.x #217

Open
zhiwilliam opened this issue Dec 10, 2022 · 2 comments
Open

Need support on Spark 3.2.x, Spark 3.3.x #217

zhiwilliam opened this issue Dec 10, 2022 · 2 comments

Comments

@zhiwilliam
Copy link

I found currently, ZparkIO Test cannot work with Spark 3.2.x and Spark 3.3.x

I simply created a project from demo (sbt new leobenkel/zparkio.g8), fixed some issues to make it work. I need to read a delta table so I added delta-core in my sbt file.

When I run the application I found delta-core is not working with 3.1.x spark. And if I upgrade to 3.2.x and run again I found the following issue.

[error] stack trace is suppressed; run 'last update' for the full output
[error] stack trace is suppressed; run 'last ssExtractDependencies' for the full output
[error] (update) lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts:
[error] https://repo1.maven.org/maven2/org/apache/avro/avro-mapred/1.10.2/avro-mapred-1.10.2-hadoop2.jar: not found: https://repo1.maven.org/maven2/org/apache/avro/avro-mapred/1.10.2/avro-mapred-1.10.2-hadoop2.jar
[error] (ssExtractDependencies) lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts:
[error] https://repo1.maven.org/maven2/org/apache/avro/avro-mapred/1.10.2**/avro-mapred-1.10.2-hadoop2.ja**r: not found: https://repo1.maven.org/maven2/org/apache/avro/avro-mapred/1.10.2/**avro-mapred-1.10.2-hadoop2.jar**
[error] Total time: 4 s, completed Dec 10, 2022, 9:37:14 AM

@leobenkel
Copy link
Owner

I will look into it

leobenkel added a commit that referenced this issue Dec 19, 2022
leobenkel added a commit that referenced this issue Dec 19, 2022
* Upgrade to latest versions

Address #217 and #213

* Remove bad options for some scala version

* Simplify ci-cd

* Remove other conflicting scoptions

* Better system for the scalaCoptions

* More fixes

* More fixes

* Try again

* Add sbt cache
@leobenkel
Copy link
Owner

Done. Please confirmed that it is working :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants