[SPARK-21411][YARN] Lazily create FS within kerberized UGI to avoid token acquiring failure

[jerryshao]

What changes were proposed in this pull request?

In the current YARNHadoopDelegationTokenManager, FileSystem to which to get tokens are created out of KDC logged UGI, using these FileSystem to get new tokens will lead to exception. The main thing is that Spark code trying to get new tokens from the FS created with token auth-ed UGI, but Hadoop can only grant new tokens in kerberized UGI. To fix this issue, we should lazily create these FileSystem within KDC logged UGI.

How was this patch tested?

Manual verification in secure cluster.

CC @vanzin @mgummelt please help to review, thanks!

[SparkQA]

Test build #79601 has finished for PR 18633 at commit 9a4f012.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[jerryshao]

Jenkins, retest this please.

[vanzin]

Cache the result of this call since it might be used again below?

[vanzin]

I think this closure needs to take a Configuration as paramater.

If you look at the old code, when renewing tokens, the file system list uses AMCredentialRenewer.freshHadoopConf as the configuration, not the ApplicationMaster configuration as your code is doing. That sounds more correct.

[jerryshao]

Ohh, I see, that looks like another issue, let me change the code.

[SparkQA]

Test build #79626 has finished for PR 18633 at commit 9a4f012.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #79684 has finished for PR 18633 at commit 95988c1.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[jiangxb1987]

LGTM

[vanzin]

Merging to master.