[SPARK-48611][CORE] Log TID for input split in HadoopRDD and NewHadoopRDD #46966
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
Author
|
|
||
| private val split = theSplit.asInstanceOf[HadoopPartition] | ||
| logInfo(log"Input split: ${MDC(INPUT_SPLIT, split.inputSplit)}") | ||
| logInfo(log"Task (TID ${MDC(TASK_ID, context.taskAttemptId())}) input split: " + |
Member
Author
There was a problem hiding this comment.
context.taskAttemptId() is task id, this was clarified in #45834 (comment)
gengliangwang
approved these changes
Jun 15, 2024
Member
|
Thanks, merging to master |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Log
TIDfor "input split" inHadoopRDDandNewHadoopRDDWhy are the changes needed?
This change should benefit both structured logging enabled/disabled cases.
When structured logging is disabled, and executor cores > 1, the logs of tasks are mixed in stdout, something like
it's hard to say which file is read by which task because they run in parallel.
If something goes wrong, the log prints
TIDand exception stack trace, the error may related to the input data, sometimes thatexception messageis clear enough to show which file that input data comes from, but sometimes not, in the latter case, the current log is not clear enough to allow us to identify the bad file quickly.When structured logging is enabled, exposing TID as a LogKey makes the logs more selective.
Does this PR introduce any user-facing change?
Yes, it supplies additional information in logs.
How was this patch tested?
Review, as it only touches log contents.
Was this patch authored or co-authored using generative AI tooling?
No