Maintaining sorted/partitioned knowledge

[devin-petersohn]

It would be beneficial if we had a way to know that a given file was sorted when it was written. As is, RDDs are force-sorted and partitioned before operations that require sorted RDDs.

Implementing Issue #1175 is a part of this work.

[heuermh]

See also #188

[devin-petersohn]

@fnothaft Following up from our conversation this morning, the mix-in implementation is finally debugged. I am going to move toward working on splitting partition and sort.

Here is the working repo for those interested: https://github.com/devin-petersohn/adam/tree/partitioner

[jpdna]

@devin-petersohn - just want to make sure you have seen this thread: #651

[jpdna]

It may be useful to look at:
https://issues.apache.org/jira/browse/SPARK-8836
https://issues.apache.org/jira/browse/SPARK-2213

[devin-petersohn]

Resolved with #1324.