[SPARK-48145][CORE] Remove logDebug and logTrace with MDC in JAVA structured logging framework

[gengliangwang]

What changes were proposed in this pull request?

Since we are targeting on migration INFO/WARN/ERROR level logs to structure logging, this PR removes the logDebug and logTrace methods from the JAVA structured logging framework.

Why are the changes needed?

In the log migration PR #46390, there are unnecessary changes such as updating

logger.debug("Task {} need to spill {} for {}", taskAttemptId,
            Utils.bytesToString(required - got), requestingConsumer);

to

LOGGER.debug("Task {} need to spill {} for {}", String.valueOf(taskAttemptId),
            Utils.bytesToString(required - got), requestingConsumer.toString());

With this PR, we can avoid such changes during log migrations.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Existing UT.

Was this patch authored or co-authored using generative AI tooling?

No

[gengliangwang]

cc @panbingkun as well.

[dongjoon-hyun]

Is this consistent with SPIP, @gengliangwang ?

[dongjoon-hyun]

#46390 is not merged yet, isn't it?

In the log migration PR #46390, there are unnecessary changes such as updating

[gengliangwang]

Is this consistent with SPIP, @gengliangwang ?

Yes, the SPIP only suggests the logInfo/logWarn/logError APIs. LogDebug/LogTrace is optional. Since #46390 is not migrating the debug/trace level logs, I suggest merging this one to reduce unnecessary changes in it.

[gengliangwang]

In Scala side, the logDebug/logTrace APIs was introduced for debugging cache purposes: #45990

[dongjoon-hyun]

+1, LGTM. Thank you!

[dongjoon-hyun]

cc @cloud-fan as a release manager for Apache Spark 4.0.0-preview

[dongjoon-hyun]

Merged to master for Apache Spark 4.0.0-preview.

[dongjoon-hyun]

Thank you, @gengliangwang .

[panbingkun]

cc @panbingkun as well.

Got it.
Thanks.