SPARK-6245 [SQL] jsonRDD() of empty RDD results in exception #4971
Conversation
|
Test build #28450 has started for PR 4971 at commit
|
|
LGTM, thanks @srowen |
There was a problem hiding this comment.
This is a super nit, but I think typically we would just do Set.empty here or maybe Set.empty[(String, DataType)] if you really want to be explicit.
There was a problem hiding this comment.
No prob, will do, and if the tests succeed I'll merge.
|
Test build #28450 has finished for PR 4971 at commit
|
|
Test PASSed. |
|
Test build #28473 has started for PR 4971 at commit
|
|
Test build #28473 has finished for PR 4971 at commit
|
|
Test PASSed. |
Avoid `UnsupportedOperationException` from JsonRDD.inferSchema on empty RDD. Not sure if this is supposed to be an error (but a better one), but it seems like this case can come up if the input is down-sampled so much that nothing is sampled. Now stuff like this: ``` sqlContext.jsonRDD(sc.parallelize(List[String]())) ``` just results in ``` org.apache.spark.sql.DataFrame = [] ``` Author: Sean Owen <sowen@cloudera.com> Closes #4971 from srowen/SPARK-6245 and squashes the following commits: 3699964 [Sean Owen] Set() -> Set.empty 3c619e1 [Sean Owen] Avoid UnsupportedOperationException from JsonRDD.inferSchema on empty RDD
4 participants
Avoid
UnsupportedOperationExceptionfrom JsonRDD.inferSchema on empty RDD.Not sure if this is supposed to be an error (but a better one), but it seems like this case can come up if the input is down-sampled so much that nothing is sampled.
Now stuff like this:
just results in