[SPARK-34232][CORE] Redact SparkListenerEnvironmentUpdate event in log

[warrenzhu25]

What changes were proposed in this pull request?

Redact event SparkListenerEnvironmentUpdate in log when its processing time exceeded logSlowEventThreshold

Why are the changes needed?

Credentials could be exposed when its processing time exceeded logSlowEventThreshold

Does this PR introduce any user-facing change?

No

How was this patch tested?

Manually tested

[HeartSaVioR]

Let's reuse EventLoggingListener.redactEvent. Probably change the code in EventLoggingListener like below, and call here?

private[spark] class EventLoggingListener(

...

  private[spark] def redactEvent(
      event: SparkListenerEnvironmentUpdate): SparkListenerEnvironmentUpdate = {
    EventLoggingListener.redactEvent(sparkConf, event)
  }
}

private[spark] object EventLoggingListener extends Logging {

...

  def redactEvent(
      sparkConf: SparkConf,
      event: SparkListenerEnvironmentUpdate): SparkListenerEnvironmentUpdate = {
    // environmentDetails maps a string descriptor to a set of properties
    // Similar to:
    // "JVM Information" -> jvmInformation,
    // "Spark Properties" -> sparkProperties,
    // ...
    // where jvmInformation, sparkProperties, etc. are sequence of tuples.
    // We go through the various  of properties and redact sensitive information from them.
    val noRedactProps = Seq("Classpath Entries")
    val redactedProps = event.environmentDetails.map {
      case (name, props) if noRedactProps.contains(name) => name -> props
      case (name, props) => name -> Utils.redact(sparkConf, props)
    }
    SparkListenerEnvironmentUpdate(redactedProps)
  }
}

[HeartSaVioR]

ok to test

[HeartSaVioR]

Looks OK, but the code change is proposed by me so I'd feel comfort to wait for another review.

[HyukjinKwon]

ok to test

[dongjoon-hyun]

+1, LGTM (only one minor comment about function visibility).

[SparkQA]

Test build #134482 has started for PR 31335 at commit 618ddc4.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39066/

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39068/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39068/

[SparkQA]

Test build #134481 has finished for PR 31335 at commit ab796cf.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

Merged to master and branch-3.1.