Allow asynchronous COPY

[jacek-rzrz]

This is probably a long shot but I thought I can ask about it here anyway.

Saving a DataFrame to Redshift can be seen as a 2-phase process:

1. Writing files to S3
2. COPYing data from S3 to a Redshift table

Of course the driver program is blocked on DataFrameWriter.save() for the duration of both phases.
During phase 2 no Spark jobs are running because things happen outside the cluster, i.e. between S3 and Redshift.
In my setup phase 2 takes much longer than phase 1 and amounts to substantial amount of time.
Perhaps this is uncommon, but I don't actually need to wait for the data to materialize in Redshift to proceed with next jobs in my driver program.

I am wondering if it would be a compelling feature to allow step 2 to happen asynchronously? I would imagine that in such scenario e.g. a Future representing phase 2 could somehow be passed to client code however I don't think Data Sources API allows for something like that.

If such a feature would be of interest and if we found a sane way to do it - I would be happy to implement.

[JoshRosen]

Today you can already wrap the DataFrameWriter.save() call in a Future and it should work as you would expect because Spark and spark-redshift should be thread-safe. However, I'm guessing that this might not be exactly what you want in case you only want the second COPY phase to be asynchronous.

As you've observed, the Data Sources API doesn't really have a concept that this type of asynchrony would map onto. The only approach that I can think of would be to expose a separate set of lower-level public APIs for the various phases of spark-redshift (on top of which we'd have the existing DataFrameWriter / Reader implementations) and define objects / case-classes for holding the data that's passed between these phases (manifest paths, temp directories, etc.). This is probably going to be a big change, so if you're interested in working on something like this I'd spend some time on up-front design to build clean and stable interfaces.

[jacek-rzrz]

Hi Josh,
Yes exposing distinct phases sounds good - I would definitely be interested to work on this. Can't devote any time to it at the moment, but starting August I could certainly spend a few hours a week at least.