[SPARK-24131][PySpark] Add majorMinorVersion API to PySpark for determining Spark versions #21203
Conversation
|
This is moved from #21153 based on @jkbradley's suggestion. |
|
retest this please. |
dongjoon-hyun
left a comment
dongjoon-hyun
left a comment
There was a problem hiding this comment.
+1, LGTM except one minor comment.
| >>> version = "abc" | ||
| >>> majorMinorVersion(version) is None | ||
| True | ||
There was a problem hiding this comment.
nit. Could you remove redundant empty line?
felixcheung
left a comment
felixcheung
left a comment
There was a problem hiding this comment.
perhaps better to match python's version_info ?
https://docs.python.org/2/library/sys.html#sys.version_info
This for now is more close to |
|
Retest this please |
|
Test build #89982 has finished for PR 21203 at commit
|
|
Merged to master. |
|
ah, sorry, @jkbradley asked me to do a little change like throwing exception when can't paring input. My flight just lands now. Let me submit a small follow-up. |
|
I missed that comment too. Yea, I think it makes more sense to throw an exception - I usually use |
| return argspec | ||
|
|
||
|
|
||
| def majorMinorVersion(version): |
There was a problem hiding this comment.
Also, I think he asked to add VersionUtils class.
|
Sorry @jkbradley, I rushed it. It looked making sense when I merged this but I didn't closely check the Scala side. Will be careful next time. |
|
@jkbradley A follow-up has been submitted and merged. Please see #21211. Thanks. |
6 participants
What changes were proposed in this pull request?
We need to determine Spark major and minor versions in PySpark. We can add a
majorMinorVersionAPI to PySpark which is similar to the Scala API inVersionUtils.majorMinorVersion.How was this patch tested?
Added tests.