[MINOR][SPARKR][DOC] Add a description for running unit tests in Windows

[HyukjinKwon]

What changes were proposed in this pull request?

This PR adds the description for running unit tests in Windows.

How was this patch tested?

On a bare machine (Window 7, 32bits), this was manually built and tested.

[HyukjinKwon]

cc @sun-rui

[SparkQA]

Test build #58963 has finished for PR 13217 at commit b62b631.

• This patch fails MiMa tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #58962 has finished for PR 13217 at commit acb9363.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

CC @steveloughran for comment. I think the paragraph should start with "It is not included in the Hadoop binary releases, so .... However it is downloadable from, for example [...]"

[HyukjinKwon]

Thank you. I will wait for the comment and will fix.

[steveloughran]

I wouldn't recommend putting it under the root of the project, as that only complicates the source tree and path cleanup; an adjacent directory works. And I think you may find that HADOOP.DLL is needed in places, as there are some JNI calls related to local file access and permissions/ACLs

At some point we (the Hadoop team) may start releasing the Windows binaries direct. It's only avoided as it complicates the release process somewhat, though if it encourages testing it can only be encouraged.

I'd suggest the following text:

---

To run the SparkR unit tests on Windows, the following steps are required —assuming you are in the Spark root directory and do not have Apache Hadoop installed already:

1. cd ..
2. mkdir hadoop
3. Download the relevant Hadoop bin package from steveloughran/winutils. While these are not official ASF artifacts, they are built from the ASF release git hashes by a Hadoop PMC member on a dedicated Windows VM.
4. Install the files into hadoop\bin; make sure that winutils.exe and hadoop.dll are present.
5. Set the environment variable HADOOP_HOME to the full path to the newly created hadoop directory.
6. For further reading, consult Windows Problems on the Hadoop wiki

---

[steveloughran]

1. It's nice to see someone sitting down to deal with the windows test problem.
2. Hadoop 2.8+ will fail meaningfully here, with an exception including a link to the Hadoop wiki entry HADOOP-10775. I've put a link to the entry in my suggested changes, so people trying to get things working have a direct link to that page, so can stay current with whatever changes are made (more docs, ASF official releases), ...
3. Someone should really test the documentation by following it. I must excuse myself on the basis that my VM is set up and I don't want to break things.

[sun-rui]

wait to see if someone can test the documentation.

[SparkQA]

Test build #59051 has finished for PR 13217 at commit aa2839c.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

@sun-rui @steveloughran While it seems obviously better for someone to follow and test this, I wonder who is going to test this and leave some comments here.

I opened this PR during working on #13165, which I started to work on because #7025 is getting stale for several months. In this way, it seems nobody would want to try this.

(In that way, #13165 will also never be merged unless someone is trying to test. I would quit and close it if I should work on something which it seems will never be merged).

Maybe it would be better, at least for this one, to merge this for users/developers to follow this document to test, since it is a minor which does not affect the code base.

[sun-rui]

@HyukjinKwon, maybe we can merge this first. I will probably have a try later.
cc @shivaram

[shivaram]

@HyukjinKwon - Thanks a lot for updating the documentation and for working on #13165 - I think there are a number of R users who use Windows and having SparkR work on Windows (at least for the purposes of trying it out) is useful for the Spark project.

That said it has been hard to find people who have the time / resources to test SparkR on Windows machines. I wrote the initial instructions for building in Windows but a bunch of stuff has changed since then.

I think this documentation change LGTM and we can come back and update it if we find a problem.

Merging this to master, branch-2.0

[HyukjinKwon]

@shivaram Thank you so much.