[SPARK-47301][SQL][TESTS] Fix flaky ParquetIOSuite

[panbingkun]

What changes were proposed in this pull request?

The pr aims to fix flaky ParquetIOSuite, as follows:
https://github.com/panbingkun/spark/actions/runs/8167992505/job/22329299706

Why are the changes needed?

Only fix flaky UT ParquetIOSuite.

Through log analysis, the logic has reached the following point (possibly due to the timing of event processing):

spark/core/src/main/scala/org/apache/spark/scheduler/OutputCommitCoordinator.scala

Lines 161 to 164 in 5876162

	sc.foreach(_.dagScheduler.stageFailed(stage, s"Authorized committer " +
	s"(attemptNumber=$attemptNumber, stage=$stage, partition=$partition) failed; " +
	s"but task commit success, data duplication may happen. " +
	s"reason=$reason"))

spark/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala

Line 2841 in 5876162

new SparkException(s"Job aborted due to stage failure: $reason", cause = exception.orNull)

Does this PR introduce any user-facing change?

No.

How was this patch tested?

• Manually test.
• Pass GA.

Was this patch authored or co-authored using generative AI tooling?

No.

[dongjoon-hyun]

+1, LGTM.

[yaooqinn]

Merged to master.
Thank you, @panbingkun & @dongjoon-hyun

[cloud-fan]

I think #46562 is a better fix, can we revert this workaround now?

[yaooqinn]

+1 @cloud-fan

Since the LOCs have been moved to ParquetIOWithoutOutputCommitCoordinationSuite, we need a followup for reverting

[panbingkun]

+1 @cloud-fan

Since the LOCs have been moved to ParquetIOWithoutOutputCommitCoordinationSuite, we need a followup for reverting

Let me to do it.
#46577