Skip to content

Commit

Permalink
made jsonl writes use gzip compression. (#593)
Browse files Browse the repository at this point in the history
  • Loading branch information
mdellabitta authored Nov 12, 2024
1 parent 6c416d5 commit 3b53fc9
Showing 1 changed file with 1 addition and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ trait JsonlExecutor extends Serializable {
// This should always write out as #text() because if we use #json() then the
// data will be written out inside a JSON object (e.g. {'value': <doc>}) which is
// invalid for our use
indexRecords.write.text(outputPath)
indexRecords.write.option("compression", "gzip").text(outputPath)

// Create and write manifest.

Expand Down

0 comments on commit 3b53fc9

Please sign in to comment.