[SPARK-47602][CORE] Resource managers: Migrate logError with variables to structured logging framework #45808

panbingkun · 2024-04-02T03:16:39Z

What changes were proposed in this pull request?

The pr aims to:

migrate logError in module Resource managers with variables to structured logging framework.
add some test case to *LoggingSuite
add new UT MDCSuite
make the type of value from String to Any( so that when using MDC, some code can be omitted, eg: String.valueOf(...) , x.toString)

Why are the changes needed?

To enhance Apache Spark's logging system by implementing structured logging.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass GA.
Add & update new UT.

Was this patch authored or co-authored using generative AI tooling?

No.

…s to structured logging framework

panbingkun · 2024-04-02T03:26:24Z

common/utils/src/main/scala/org/apache/spark/internal/Logging.scala

 * part of the ThreadContext.
 */
-case class MDC(key: LogKey.Value, value: String)
+case class MDC(key: LogKey, value: Any)


Make the type of value from String to Any, so that when using MDC, some code can be omitted, eg: String.valueOf(...) , x.toString

panbingkun · 2024-04-02T03:28:23Z

cc @gengliangwang

panbingkun · 2024-04-02T03:39:46Z

common/utils/src/test/scala/org/apache/spark/util/StructuredLoggingSuite.scala


+  def expectedPatternForBasicMsg(level: Level): String
+
+  def expectedPatternForBasicMsgWithException(level: Level): String


Add a new test for basicMsg + Exception

Nit: Let's add comments about each pattern is about

gengliangwang · 2024-04-02T04:36:23Z

common/utils/src/main/scala/org/apache/spark/internal/LogKey.scala

 */
 object LogKey extends Enumeration {
-  val EXECUTOR_ID, MIN_SIZE, MAX_SIZE = Value
+  val APPLICATION_ID, APPLICATION_STATE, BUCKET, CONTAINER_ID, EXECUTOR_ID, POD_ID = Value


Each enum can take one line.
val APPLICATION_ID = Value
val APPLICATION_STATE = Value

gengliangwang · 2024-04-02T04:36:28Z

common/utils/src/main/scala/org/apache/spark/internal/LogKey.scala

+  val MIN_SIZE, MAX_SIZE, MAX_EXECUTOR_FAILURES = Value
+  val EXIT_CODE = Value
+
+  type LogKey = Value


why do we need this?

Because I want to use 'xxx: LogKey' instead of 'xxx: LogKey.Value' to define a type, it feels like it's a personal preference. Should we restore it?

spark/core/src/main/scala/org/apache/spark/scheduler/TaskLocality.scala

Lines 23 to 32 in c3da260

object TaskLocality extends Enumeration {

// Process local is expected to be used ONLY within TaskSetManager for now.

val PROCESS_LOCAL, NODE_LOCAL, NO_PREF, RACK_LOCAL, ANY = Value

type TaskLocality = Value

def isAllowed(constraint: TaskLocality, condition: TaskLocality): Boolean = {

condition <= constraint

}

}

gengliangwang · 2024-04-02T04:37:42Z

common/utils/src/test/scala/org/apache/spark/util/MDCSuite.scala

+
+  test("check MDC message") {
+    val log = log"This is a log, exitcode ${MDC(EXIT_CODE, 10086)}"
+    assert(log.message === "This is a log, exitcode 10086")


Let's verify the returned map as well

gengliangwang · 2024-04-02T04:47:19Z

...netes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsAllocator.scala

 import org.apache.spark.deploy.k8s.KubernetesConf
 import org.apache.spark.deploy.k8s.KubernetesUtils.addOwnerReference
-import org.apache.spark.internal.Logging
+import org.apache.spark.internal.{Logging, LogKey => LK, MDC}


Why alias as LK? LogKey is fine

gengliangwang · 2024-04-02T04:52:15Z

@panbingkun Thanks for the work. LGTM except some minor comments.

panbingkun · 2024-04-02T05:30:18Z

I will update soon so that we can proceed smoothly with the following work.

panbingkun · 2024-04-02T06:06:04Z

@panbingkun Thanks for the work. LGTM except some minor comments.

Done.

panbingkun · 2024-04-02T08:48:41Z

resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala

          } else {
-            logError(s"Driver terminated with exit code ${exitCode}! Shutting down. $remoteAddress")
+            logError(log"Driver terminated with exit code ${MDC(EXIT_CODE, exitCode)}! " +
+              log"Shutting down. ${MDC(REMOTE_ADDRESS, remoteAddress)}")


@gengliangwang
Perhaps it would be more reasonable to write this in this way

logError(log"Driver terminated with exit code ${MDC(EXIT_CODE, exitCode)}! " + log"Shutting down. $remoteAddress")

Not every variable needs to be recorded in the field content of JSON
Unfortunately, we currently do not support this syntax as above, it will fail to compile.
So I submitted a separate PR to address this issue: #45813

@panbingkun Let's put all the variables into the context during the migration. If the context is too verbose, we can configure log4j to only show a subset of keys later.
Otherwise, there will be back-and-forth during the migration. remoteAddress can be helpful for debugging here.

gengliangwang · 2024-04-02T17:03:57Z

Thanks, merging to master

pan3793 · 2024-04-03T05:52:19Z

...netes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsAllocator.scala

      case e: Exception =>
-        logError(s"Cannot get the creationTimestamp of the pod: ${state.pod}", e)
+        logError(log"Cannot get the creationTimestamp of the pod: " +
+          log"${MDC(LogKey.POD_ID, state.pod)}", e)


IMO POD_ID is not a suitable key here. state.pod actually outputs a whole Pod Spec, and there is no such a "pod id" concept in the K8s, in practice, we use namespace and pod name to identify a pod.

@pan3793 Thanks for the suggestion. Do you mind creating a PR to improve this?

I would like to have a try, it may take a few hours to learn this new log framework.

The inital PR of the framework is in #45729. Thank you in advance!

@pan3793 Thank you very much for correcting this issue.

[SPARK-47602][CORE] Resource managers: Migrate logError with variable…

6edda2b

…s to structured logging framework

github-actions bot added KUBERNETES YARN labels Apr 2, 2024

panbingkun commented Apr 2, 2024

View reviewed changes

panbingkun marked this pull request as ready for review April 2, 2024 03:28

panbingkun commented Apr 2, 2024

View reviewed changes

gengliangwang reviewed Apr 2, 2024

View reviewed changes

panbingkun added 2 commits April 2, 2024 14:04

fix

8ed977f

Merge branch 'master' into SPARK-47602

98669a2

fix

bf70abb

panbingkun commented Apr 2, 2024

View reviewed changes

panbingkun mentioned this pull request Apr 2, 2024

[SPARK-47688][CORE] Support three methods of the log concatenation in the structured logging framework #45813

Closed

Merge branch 'master' into SPARK-47602

23ce760

gengliangwang approved these changes Apr 2, 2024

View reviewed changes

gengliangwang closed this in db0975c Apr 2, 2024

pan3793 reviewed Apr 3, 2024

View reviewed changes

pan3793 mentioned this pull request Apr 3, 2024

[SPARK-47602][CORE][K8S][FOLLOWUP] Improve structure logging for isExecutorIdleTimedOut #45849

Closed


		def expectedPatternForBasicMsg(level: Level): String

		def expectedPatternForBasicMsgWithException(level: Level): String

	object TaskLocality extends Enumeration {
	// Process local is expected to be used ONLY within TaskSetManager for now.
	val PROCESS_LOCAL, NODE_LOCAL, NO_PREF, RACK_LOCAL, ANY = Value

	type TaskLocality = Value

	def isAllowed(constraint: TaskLocality, condition: TaskLocality): Boolean = {
	condition <= constraint
	}
	}

[SPARK-47602][CORE] Resource managers: Migrate logError with variables to structured logging framework #45808

[SPARK-47602][CORE] Resource managers: Migrate logError with variables to structured logging framework #45808

Uh oh!

Conversation

panbingkun commented Apr 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

panbingkun commented Apr 2, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gengliangwang commented Apr 2, 2024

Uh oh!

panbingkun commented Apr 2, 2024

Uh oh!

panbingkun commented Apr 2, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gengliangwang commented Apr 2, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gengliangwang Apr 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

panbingkun commented Apr 2, 2024 •

edited

Loading

gengliangwang Apr 3, 2024 •

edited

Loading