-
Notifications
You must be signed in to change notification settings - Fork 138
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix compatibility issues with the Databricks runtime #506
Comments
Hi. I am running into this issue, specifically
when running the example UDF code found on the Using ScalaPB with Spark page. This is similar to what is reported on stackoverflow here. For reference I am using
I reached out to Databricks internal support and here is their response:
so this appears related to #300 . They also provided an alternative:
I tried it with @dsabanin 's fix linked above and confirmed it does avoid the error, although I suspect that particular patch would limit the frameless library's usability with "vanilla" Spark. Would frameless be open to refactoring the expressions to extend |
… often needs to be implemented anyway
fyi - so far I've not found issues using doCodeGen instead, Quality cross compiles from 2.4 with many supported Databricks LTS versions. There are lots of hacks needed to work with DBRs at that level, often having to reproduce interfaces specific the kind the DBRs use, but doCodeGen works throughout. I'm not using typed datasets or udfs in the project so I've not been hit by this. In OSS genCode wraps doGenCode checking for dupes, adding null checks etc. Frameless wise I don't see any reason doGenCode can't just be used instead, dataset tests pass etc. I've raised pr 700 for that change. |
Fix is shown here: dsabanin@1ce8f0b
Thanks to @dsabanin for the fix!
The text was updated successfully, but these errors were encountered: