[SPARK-46759][SQL][AVRO] Codec xz and zstandard support compression level for avro files #44786
Conversation
…evel for avro files
| .intConf | ||
| .checkValues((1 to 9).toSet + Deflater.DEFAULT_COMPRESSION) | ||
| .createWithDefault(Deflater.DEFAULT_COMPRESSION) | ||
| .createOptional |
There was a problem hiding this comment.
Why do we need to remove the default Deflater.DEFAULT_COMPRESSION here?
There was a problem hiding this comment.
Oh, nvm. I found it goes to DEFLATE(DataFileConstants.DEFLATE_CODEC, true, CodecFactory.DEFAULT_DEFLATE_LEVEL),.
dongjoon-hyun
left a comment
dongjoon-hyun
left a comment
There was a problem hiding this comment.
This is really a nice improvement. Thank you, @yaooqinn .
|
Merged to master for Apache Spark 4.0.0. |
|
Thank you very much @dongjoon-hyun |
| val level = sqlConf.getConfString(s"spark.sql.avro.$codecName.level", | ||
| compressed.getDefaultCompressionLevel.toString) | ||
| logInfo(s"Compressing Avro output using the $codecName codec at level $level") | ||
| val s = if (compressed == ZSTANDARD) "zstd" else codecName |
There was a problem hiding this comment.
It seems we should put logInfo(s"Compressing Avro output using the $codecName codec at level $level") after val s = if (compressed == ZSTANDARD) "zstd" else codecName.
There was a problem hiding this comment.
@beliefer May I ask your reason? For me, it's not required because Avro's real codec name is zstandard instead of zstd .
public static final String ZSTANDARD_CODEC = "zstandard";
SPARK REPO
There was a problem hiding this comment.
Got it. zstd only used for avro.mapred.zstd.level.
@dongjoon-hyun Thank you for your explanation.
3 participants
What changes were proposed in this pull request?
This PR introduces 2 keys in the form of 'spark.sql.avro.$codecName.level' just like the existing 'spark.sql.avro.deflate.level' for standard and xz codec. W/ this patch, users are able to play the trade-off between the speed and compression ratio when they use AVRO compressed by zstd or xz.
Why are the changes needed?
Avro supports compression level for deflate, xz and zstd, but we have only supported deflate.
Does this PR introduce any user-facing change?
yes, new configurations added
How was this patch tested?
new tests
Was this patch authored or co-authored using generative AI tooling?
no