Better handling of the partitions

[JulienPeloton]

This is linked to the issue #6, but somewhat different.
The way it is done now is that we have partition size roughly equal to HDFS block size (~128 MB). Ideally partition size should follow resource of the cluster (typically 2-3x the number of cores or executors in use).
I guess repartitioning at the very end would be very costly though... Need to investigate at a lower level then.