[SPARK-27750][CORE] Standalone scheduler - ability to prioritize applications over drivers, many drivers act like Denial of Service #27697
Conversation
…ns over drivers, many drivers act like Denial of Service
|
Can one of the admins verify this patch? |
Ngone51
left a comment
Ngone51
left a comment
There was a problem hiding this comment.
How about limiting the number of concurrent running drivers? Personally, I feel it's more convenient for user to set # of driver rather than the cores. WDYT?
|
@Ngone51 but each driver can have different amount of cores |
Yeah, but for such case, many drivers race for the reserved core concurrently, how can we know which drivers win the race and decide the amount of cores for them? |
|
@Ngone51 as long as some cores are not used by any driver, then i don't mind which drivers got in first. I just want some cores that can only be consumed by apps |
When we limit the number of running drivers, we'll definitely have reserved cores for the apps. |
|
@Ngone51 that is worse, one day you could have 10 drivers each taking 1 core, another day you could have 5 drivers each taking 4 cores (in this case if u limited on 10 drivers on both days, all cores would be used on 2nd day). So it does not work well with diff sized drivers. Cores is simple/uniform way. Secondly, if I want to have scheduled scaleup of spark cluster, specifying a portion of cores to not use still works, whereas fixing a max # drivers does not work as it limits ability to take advantage of larger spark cluster |
|
What's blocking you from choosing the client deploy mode? Using the client deploy mode would let you launch the driver program locally and thus no need to worry about drivers involved into cluster resource contentions. |
|
Speaking of the proposed improvement, it does somehow migrates the issue by allowing you to launch at least a few executors, but it would probably still be the case that not enough cluster resources are going to executors, thus many drivers still need to wait for free slots to launch their pending tasks. Without other considerations, I think you should give client deploy mode a try (or you already have other issues thus not able to choose the client mode) ? |
|
@jiangxb1987 client deploy mode is subject to limits of single master machine. if I want to run 200+ spark-submits in parallel then the master machine must have a enough memory to support all those drivers. With my change few executors is all I need, but even I can reserve 100s of cores for just apps with this new config |
|
Sorry I don't have enough context to understand your use case, but submitting 200+ applications at the same time to a Spark cluster is something I'm not expecting. Basically I would expect a lot less applications, and each application can submit a few jobs, thus we don't really need to launch that much drivers. |
|
i am doing a data lake, where 10000s of files of different schemas get ingested daily, client mode does not handle the concurrency/scale i need. Also I use Spark REST API (spark.master.rest.port) which does not support client mode. what is the reluctance to merging this? |
|
can u pls merge @dongjoon-hyun ? |
| .createWithDefault(Int.MaxValue) | ||
|
|
||
| val CORES_RESERVED_FOR_APPS = ConfigBuilder("spark.deploy.coresReservedForApps") | ||
| .version("2.4.6") |
There was a problem hiding this comment.
Hi, @tooptoop4 . We don't backport a feature.
The feasible next version is 3.1.0.
|
The proposed improvement is just not necessary. I don't see any need to submit 200+ applications, you could start one application and submit multiple jobs. |
| launched = true | ||
| val allFreeCores = shuffledAliveWorkers.map(_.coresFree).sum | ||
| val forDriversFreeCores = math.max(allFreeCores - coresReservedForApps, 0) | ||
| if (forDriversFreeCores > 0) { |
There was a problem hiding this comment.
Could you make a test case for this?
dongjoon-hyun
changed the title
|
@jiangxb1987 ur suggestion is don't scale? |
|
What do you mean by scale? You don't need 200 drivers, you can still launch one application and submit your jobs, this way your workload should work. |
|
|
I‘m -1 to this change, because it's trying to resolve an issue that doesn't even exist. |
|
Thank you for your proposal, @tooptoop4. However, according to the above discussion, I'll close this PR. Thank you, @tooptoop4 , @Ngone51 , @jiangxb1987 . |
5 participants
What changes were proposed in this pull request?
Don't allow drivers to use all the cores in spark standalone scheduler. Reserve some cores for apps
Why are the changes needed?
Airflow is triggering many Spark jobs in parallel, spark standalone gets 'full' with drivers and can never execute apps as drivers took all the cores. The spark cluster essentially becomes stuck and never completes any drivers (as no cores avail to run the associated apps), never frees up cores. Manual intervention is needed to either kill some drivers (but airflow can retry sending them), or force scale up of the spark cluster by registering more workers (than ASG MaxSize).
Does this PR introduce any user-facing change?
By default no, they must opt-in by explicitly setting config value > 0
How was this patch tested?
Running in prod for 3 months