[SPARK-52060][SQL] Make OneRowRelationExec node

[richardc-db]

What changes were proposed in this pull request?

creates a new OneRowRelationExec node, which is more or less a copy of the RDDScanExec node.

We want a dedicated node because this helps make it more clear when a one row relation, i.e. for patterns like SELECT version() is used.

Why are the changes needed?

this makes it more clear in the code that a one row relation is used and allows us to avoid checking the hard coded "OneRowRelation" string when pattern matching.

Does this PR introduce any user-facing change?

yes, the plan will now be OneRowRelationExec rather than RDDScanExec. The plan string should be the same, however.

How was this patch tested?

added UTs

Was this patch authored or co-authored using generative AI tooling?

[cloud-fan]

How is this different from the default implementation?

[richardc-db]

the default implementation returns Scan OneRowRelation, while the existing implementation (using RDDScan) returns Scan OneRowRelation[]. I figured we shouldn't change this in the off chance that someone is relying on it.

[richardc-db]

there are failures like

===== POSSIBLE THREAD LEAK IN SUITE o.a.s.sql.errors.QueryExecutionErrorsSuite, threads: rpc-boss-1245-1 (daemon=true) =====


[info] org.apache.spark.sql.errors.QueryExecutionErrorsSuite *** ABORTED *** (37 milliseconds)
[info]   java.lang.IllegalStateException: Shutdown hooks cannot be modified during shutdown.
[info]   at org.apache.spark.util.SparkShutdownHookManager.add(ShutdownHookManager.scala:212)

which i also see in other PRs such as here, so i think the failures are unrelated

[cloud-fan]

I think we can do this

private val rdd = {
  val proj = UnsafeProjection.create(schema)
  val emptyRow = proj(InternalRow.empty)
  session.sparkContext.parallelize(Seq(emptyRow), 1)
}

[cloud-fan]

then def doExecute() can just return this RDD

[richardc-db]

hmm, sure i implemented this - but because we still want to increment the number of output rows at the time when the row is actually processed, i did not end up simply returning rdd from doExecute()... lmk if you think there a better way.

[cloud-fan]

If we do https://github.com/apache/spark/pull/50849/files#r2097521541 , then this can be false.

[cloud-fan]

@richardc-db can you do a rebase to re-trigger the CI? We fixed an OOM test issue recently.

[richardc-db]

hmm, @cloud-fan, seems like this causes a test to fail with org.apache.spark.sql.CacheTableInKryoSuite. The error is

java.io.IOException: com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Class is not registered: org.apache.spark.sql.catalyst.expressions.UnsafeRow[]

I'm guessing this is because the inputRDD has unsafe rows? The test passes after undoing the change described here... do you think we should undo this change and go back to adding an unsafe projection in doExecute, like so:

protected override def doExecute(): RDD[InternalRow] = {
    val numOutputRows = longMetric("numOutputRows")
    rdd.mapPartitionsWithIndexInternal { (_, iter) =>
      val proj = UnsafeProjection.create(schema)
      iter.map { r =>
        numOutputRows += 1
        proj(r)
      }
    }
  }

and setting createUnsafeProjection: Boolean = true

[cloud-fan]

@richardc-db let's change it back, didn't realize this serde issue...

[richardc-db]

@cloud-fan sounds good! I switched it back and seems like tests pass now

[cloud-fan]

thinking about it more, I think we can avoid serializing any row:

session.sparkContext.parallelize(Nil, 1).mapPartitionsInternal { _ =>
  val proj = UnsafeProjection.create(Seq.empty[Expression])
  Iterator(proj.apply(InternalRow.empty))
}

[cloud-fan]

Now the unsafe row is generated on the fly at the worker side, no serialization is needed.

[richardc-db]

done, thanks for the help! ended up doing

private val rdd: RDD[InternalRow] = {
    val numOutputRows = longMetric("numOutputRows")
    session
      .sparkContext
      .parallelize(Seq(InternalRow()), 1)
      .mapPartitionsInternal { _ =>
        val proj = UnsafeProjection.create(Seq.empty[Expression])
        Iterator(proj.apply(InternalRow.empty)).map { r =>
          numOutputRows += 1
          r
        }
      }
  }

to ensure the metrics are filled properly.

[cloud-fan]

LGTM except for one idea to improve the perf: https://github.com/apache/spark/pull/50849/files#r2107403483

[cloud-fan]

thanks, merging to master!