Fix order clause empty df handling #394
Conversation
CanBerker
added
Review Required
On hold
Question
and removed
Review Required
On hold
labels
CanBerker
force-pushed
the
fixOrderClauseEmptyDFHandling
branch
2 times, most recently
from
d2b1971 to
212e8cb
Compare
| @@ -187,6 +187,9 @@ private void setAllLocalResults() { | |||
| throw new SparksoniqRuntimeException("Invalid orderby clause."); | |||
| } | |||
| Dataset<Row> df = _child.getDataFrame(context, getProjection(parentProjection)); | |||
| if (df.count() == 0) { | |||
| return df; | |||
| } | |||
Here I think performance measurements would tell which is better. For an empty DF, the closures would not be called anyway, so that the count() action may be more costly than the gain. Maybe you could open an issue on this? What is the current status of empty DFs on other clauses? Are there any crashes? |
|
Fortunately, there were no crashes with the other clauses. I have generated the issue for performance investigations of empty DFs for later. |
CanBerker
force-pushed
the
fixOrderClauseEmptyDFHandling
branch
from
212e8cb to
6cd0689
Compare
Based on #393. This PR introduces a tiny change that fixes the crashes caused by empty dataframe being passed as input to order by clause. This would result in an empty sql statement which causes the crash.
Tests added.
@ghislainfourny Should the short circuit mentioned above (seen in OrderByClauseSparkIterator.java file) be added to other FLWOR clauses as well? It could save a bit of unnecessary computation and querying when operating on empty sequences. However this introduces an additional count() action which might be costly.