[SPARK-7771] [SPARK-7779] Dynamic allocation: lower default timeouts further

[andrewor14]

The default add time of 5s is still too slow for small jobs. Also, the current default remove time of 10 minutes seem rather high. This patch lowers both and rephrases a few log messages.

[andrewor14]

@sryza

[sryza]

This seems like a good idea to me. I could even support going lower (e.g. hundreds of ms) on the add time, given that we now cap at the number of executors needed to satisfy tasks and cancel pending requests.

[SparkQA]

Test build #33186 has finished for PR 6301 at commit 3320710.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[andrewor14]

I just tried this out on a real cluster and it works well. The output from ExecutorAllocationManager are now more understandable:

INFO ExecutorAllocationManager: Lowering target number of executors to 305 because not all requests are actually needed (previously 307)
INFO ExecutorAllocationManager: Lowering target number of executors to 301 because not all requests are actually needed (previously 305)
INFO ExecutorAllocationManager: Lowering target number of executors to 298 because not all requests are actually needed (previously 301)
...

Before this patch, we were getting a bunch of Requesting 0 new executor(s) (see SPARK-7779).

[SparkQA]

Test build #33204 has finished for PR 6301 at commit 5fcd3eb.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #33205 has finished for PR 6301 at commit 2811492.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[andrewor14]

@sryza It's good that we support it being that low but I'd rather be a little conservative for default values.

EDIT: actually we don't support it being in milliseconds currently because we cast the time to seconds, so this is the lowest it can ever get without some refactoring.

[sryza]

What's the rationale behind bumping this up to info? It seems like there will be common situations (any time that there are sustained pending tasks but we've hit our max) where this will end up printing a log every second. Which is a lot of noise.

[andrewor14]

oops I'm not sure that was an intentional change.

[andrewor14]

though at some point it might be good to log it in a limited way. It will introduce another variable but at least the user will know when s/he hits the cap.

[SparkQA]

Test build #33280 has finished for PR 6301 at commit 6d614a6.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[pwendell]

LGTM - makes sense to lower these thresholds.

[andrewor14]

Thanks @pwendell @sryza. Merging into master and 1.4.