[SPARK-3596][YARN]Support changing the yarn client monitor interval #5292
Conversation
There was a problem hiding this comment.
I would change this to something like spark.yarn.client.amPollInterval.
There was a problem hiding this comment.
Thank you, but as it is the client to get the application report from the RM, so maybe "spark.yarn.client.progress.pollinterval" is better.
There was a problem hiding this comment.
My main concern is that we don't really have a notion of "progress" in Spark on YARN. In MapReduce, progress traditionally referred to how far along the app was, but Spark doesn't provide a similar notion. YARN itself does have a notion of app progress, which is not related to what's here.
There was a problem hiding this comment.
Yeah, you are right. In PR #5305 I use the client.monitorApplication, then we can use "spark.yarn.report.interval" to changing the yarn client monitor interval. Thank you.
There was a problem hiding this comment.
OK, is this superseded by #5305 ? it sounds like you have two different changes related to SPARK-3596 and I'm wondering whether they are mutually exclusive, or one or both is preferred, or what.
There was a problem hiding this comment.
Up to your judgment; you're able to open and close PRs.
|
test this please |
|
Jenkins, test this please |
1. YarnClientSchedulerBack.asyncMonitorApplication use Client.monitorApplication so that commonize the monitor logic 2. Support changing the yarn client monitor interval, see #5292 3. More details see discussion on #3143 Author: unknown <l00251599@HGHY1L002515991.china.huawei.com> Author: Sephiroth-Lin <linwzhong@gmail.com> Closes #5305 from Sephiroth-Lin/SPARK-4346_3596 and squashes the following commits: 47c0014 [unknown] Edit conflicts 52b29fe [unknown] Interrupt thread when we call stop() d4298a1 [unknown] Unused, don't push aaacb42 [Sephiroth-Lin] don't wrap the entire block in the try ee2b2fd [Sephiroth-Lin] update 6483a2a [unknown] Catch exception 6b47ff7 [unknown] Update code 568f46f [unknown] YarnClientSchedulerBack.asyncMonitorApplication should be common with Client.monitorApplication
4 participants
No description provided.