Add temp_format argument to pack_partitions_to_parquet #22
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds a new argument to
DaskGeoDataFrame.pack_partitions_to_parquet
namedtemp_format
. This argument may be set to a format string containing a{partition}
replacement field. If provided, this string is formatted with the output partition number to generate the temporary directory path for that partition.For example
temp_format="/tmp/spatial/part-{partition}"
would create temporary directories:/tmp/spatial/part-0
/tmp/spatial/part-1
/tmp/spatial/part-2
...
The
temp_format
string may also contain a{uuid}
replacement field. If provided this will be replaced by a randomly generated UUID string. This makes it possible to reuse the sametemp_format
string in multiple simultaneous jobs without conflict.