[SPARK-5655] Don't chmod700 application files if running in YARN

[growse]

As per SPARK-5655, recently committed code chmod 700s all application files created on the local fs by a spark executor. This is both unnecessary and broken on YARN, where files created in the nodemanager's working directory are already owned by the user running the job and the 'yarn' group. Group read permission is also needed for the auxiliary shuffle service to be able to read the files, as this is running as the 'yarn' user.

[AmplabJenkins]

Can one of the admins verify this patch?

[growse]

Apologies, just seen that I've PR'd this onto branch-1.3 instead of master. Not sure if it can be easily moved.

[srowen]

I'd open a new PR vs master. The logic here is getting a little hairy, although it's not crazy. How about just setting permission to 750?

[growse]

Will do another PR.

Re: 750 - this isn't supported by the JDK, because group permissions are Posix-specific, so you'd need logic to say:

if (linux) { setPosixGroupReadable(); }

Also, if you explicitly set the permissions, it has the effect of overruling the setgid flag on the parent directory and you end up with the same group as the user. On YARN, the desired permissions of the files should be:

-rw-r-----    containeruser    yarn

The group is set from the setgid flag on the parent directories. Any explicit chmod from spark overrides this and you end up with a file that's chown'd containeruser:containeruser. Only way around this would be to somehow know the group that the YARN nodemanager process is running as, but that would start to get messy I think.

[growse]

Closing and re-opening on master as requested. Now under PR #4509.