Added config option to enable parallel deletes for vacuum command

[Kimahriman]

Resolves #395

#416 hasn't been updated in over four months, and it would be a verify useful feature for us to have, so I took my own stab at it.

• A new config value is added vacuum.parallelDelete.enabled that defaults to false
• I updated the default behavior to be coalesce to 1 instead of iterate on the driver so that you can see something being done by spark in the UI/console instead of it just sitting there. I'm not sure if there's a reason this would cause issues, so happy to revert this back if you think it should be.
• If vacuum.parallelDelete.enabled is set to true, it maintains the existing partitions from the diff calculation. Because this is the result of a join, your partitions are then based off your spark.sql.shuffle.partitions. So your parallelism will be min(number of executors, shuffle partitions), and you can tweak your shuffle partitions if you want more/less parallelism

I removed the delete static method because the number of parameters that had to be passed to it made it seem like too much. Happy to move that back if that's not preferred.

Also happy to make any updates to the name or description of the new config.

[jose-torres]

I think it's best to leave the default behavior to iterate on the driver and maintain the delete static method.

Other than that, looks good! Can you add a unit test to the vacuum test suite with the flag enabled?

[Kimahriman]

I think it's best to leave the default behavior to iterate on the driver and maintain the delete static method.

Other than that, looks good! Can you add a unit test to the vacuum test suite with the flag enabled?

Oh duh, I couldn't think of a way to unit test the behavior but I guess just making sure it at least works with the flag enabled would be good.

[Kimahriman]

I reverted back to the static method and delete on driver in the default case. Also added a basic test running a vacuum with the config enabled.

[itaydemri]

Hi, Are you going to merge this soon?
It should help me a lot!

[swapnil-chougule]

Hey Team,
It would be better if this change get merged soon
It is helpful & waiting for it.

[jose-torres]

We're working on getting this merged. Sorry for the delay!

[itaydemri]

Thanks for closing this!
Any plans to merge it to 0.6?

[zsxwing]

Any plans to merge it to 0.6?

In general, we don't backport new features.