[SPARK-7135][SQL] DataFrame expression for monotonically increasing IDs.

[rxin]

No description provided.

[SparkQA]

Test build #30959 has finished for PR 5709 at commit a7136cb.

• This patch fails Python style tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[pwendell]

Could it be confusing to users that the ID associated with each record might be different on stage or task retries? The fact that ordering within a partition is not deterministic has caused people some concern in the past, and I wonder if this could sort of lead to more confusion since you are giving some sort of ordering semantics.

[rxin]

partition id doesn't change between retries, does it?

[rxin]

guaranteed TO be

[pwendell]

No, but the ordering of records in a partition can change, so you might have different identifiers for the same record across retries (unless this is only used for already sorted data... is it?).

[rxin]

Those could change in shuffle I guess, but I don't think this is creating more confusion. What we care about here is not the record ordering, but the output of this expression is monotonic increasing. That will always be true.

This is very similar to the row id idea a lot of databases have. Records in database tables also don't have ordering, unless they are sorted.

[rxin]

@pwendell you raised a very good point about ordering of records within RDDs and DataFrames. I think we should document those more clearly in the javadoc for these.

[pwendell]

@rxin yeah I just mean if I'm in a database and I run the same query twice, I will get the same row ID for the same record. Because of non determinism in the shuffle, that's not true here.

[rxin]

(That's not always true -- somebody could've deleted an index and then the scan gets turned from index scan to sequential scan, and then record ordering changed)

[pwendell]

Oh I see - I guess it doesn't matter then.

[SparkQA]

Test build #30969 has started for PR 5709 at commit a9fda0d.

[rxin]

Jenkins, retest this please.

[yhuai]

LGTM

[SparkQA]

Test build #31112 has finished for PR 5709 at commit a9fda0d.

• This patch fails MiMa tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class MonotonicallyIncreasingID() extends LeafExpression
• This patch does not change any dependencies.

[SparkQA]

Test build #31116 has finished for PR 5709 at commit 7853611.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.