[SPARK-35868][CORE] Add fs.s3a.downgrade.syncable.exceptions if not set

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR aims to add fs.s3a.downgrade.syncable.exceptions=true if it's not provided by the users.

Why are the changes needed?

Currently, event log feature is broken with Hadoop 3.2 profile due to UnsupportedOperationException because HADOOP-17597 changes the default behavior to throw exceptions by default since Apache Hadoop 3.3.1. We know that it's because EventLogFileWriters is using hadoopDataStream.foreach(_.hflush()), but this PR aims to provide the same UX across Spark distributions with Hadoop2/Hadoop 3 at Apache Spark 3.2.0.

$ bin/spark-shell -c spark.eventLog.enabled=true -c spark.eventLog.dir=s3a://dongjoon/spark-events/
...
21/06/23 17:34:35 ERROR SparkContext: Error initializing SparkContext.
java.lang.UnsupportedOperationException: S3A streams are not Syncable. See HADOOP-17597.

Does this PR introduce any user-facing change?

Yes, this will recover the existing behavior.

How was this patch tested?

Manual.

$ build/sbt package -Phadoop-3.2 -Phadoop-cloud
$ bin/spark-shell -c spark.eventLog.enabled=true -c spark.eventLog.dir=s3a://dongjoon/spark-events/
...(working)...

If the users provide the configuration explicitly, it will return to the original behavior throwing exceptions.

$ bin/spark-shell -c spark.eventLog.enabled=true -c spark.eventLog.dir=s3a://dongjoon/spark-events/ -c spark.hadoop.fs.s3a.downgrade.syncable.exceptions=false
...
21/06/23 17:44:41 ERROR Main: Failed to initialize Spark session.
java.lang.UnsupportedOperationException: S3A streams are not Syncable. See HADOOP-17597.

[dongjoon-hyun]

cc @sunchao , @steveloughran

[dongjoon-hyun]

cc @gengliangwang for Apache Spark 3.2.0.

[sunchao]

LGTM (non-binding) thanks @dongjoon-hyun !

[dongjoon-hyun]

Thank you, @sunchao !

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44756/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44756/

[gengliangwang]

LGTM

[SparkQA]

Test build #140229 has finished for PR 33044 at commit 4cbf380.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Thank you so much, @gengliangwang ! The python UT failures are irrelevant.
Merged to master for Apache Spark 3.2.0.

[HyukjinKwon]

lgtm2

[steveloughran]

thx. FWIW, given its causing trouble, do you want this to be the default in hadoop default-xml?

its there to stop people attempting to use s3 as a WAL for HBase or similar, but if applications have been treating it as a low-cost operation in general file IO, then we can just downgrade it broadly and rely on the hope that people don't do this.