[SPARK-20992][Scheduler] Add links in documentation to Nomad integration.

[barnardb]

What changes were proposed in this pull request?

Adds links to the fork that provides integration with Nomad, in the same places the k8s integration is linked to.

How was this patch tested?

I clicked on the links to make sure they're correct ;)

[AmplabJenkins]

Can one of the admins verify this patch?

[srowen]

We can't add this here because it is not supported in Spark. Actually k8s kind of shouldn't be there yet but it's being integrated now. We can put this on the 'powered by' page or you can put it on the spark packages site.

[rcgenova]

We don't see much in the way of Kubernetes pull requests against Spark. Can you elaborate on exactly what you mean by "it's being integrated now"?

[srowen]

@rcgenova I'm referring to https://issues.apache.org/jira/browse/SPARK-18278
This note was added in #17522 and I actually disagree that this should have gone in. Why not add this when it's actually in Spark? @foxish @rxin @mridulm
I think this PR here is one of the reasons, actually

[mridulm]

@srowen I agree, kubernetes doc note should also not have gone in IMO. Unfortunately I was not around to press the point.

[foxish]

We don't see much in the way of Kubernetes pull requests against Spark. Can you elaborate on exactly what you mean by "it's being integrated now"?

@rcgenova, we're working with @felixcheung, @rxin and testing infrastructure folks from RISELab - getting our CI testing setup, addressing feedback on PRs, simplifying them into appropriate reviewable chunks that can be sent up here, and stabilizing the release for the existing audience in our own repo -
https://github.com/apache-spark-on-k8s/spark/.

@srowen, @mridulm, I understand your concern here and it is definitely a valid one. When the note was added for Kubernetes, it was made clear to us that this is not meant to offer a path to an upstream merge (#17522 (review)), but to help build the consensus, community maintainership and adoption required for our own separate fork. TBH, we didn't see the doc note have a credible impact on the activity we saw on our fork. Around 2% of the visits to our docs/repo came with a spark.apache.org referer, which may validate your opinion that adding into the docs did not really meet its purpose -- even for us.

I'm in no way opposed to any cluster manager integrations (be it nomad or any other), as long as they are useful and maintainable. I just think regular emails to the spark-dev list and the SPIP/voting process were far more useful to us to building maintainership and an audience, than the documentation snippet.

[felixcheung]

+1 on powered by or similar page links.
I agree the link to a non-ASF project/fork from within the official documentation seems pre-mature.
And for sure, we should look at what changes and extensibility can be built to encourage the community to develop new cluster managers as a standalone package.

[barnardb]

I totally understand the reluctance to have non-ASF projects in a list headed by "The system currently supports…". Looking at the Powered By page, it doesn't look like the best way to help users find third-party cluster integrations. The Spark Packages site @srowen mentions seems a slightly better fit, but I don't think it would occur to me that I could look there for cluster manager integrations. Would it be appropriate to remove the Kubernetes and Nomad items from the list of supported cluster managers, and instead follow the list with a short line that links to these integrations but makes their unsupported status clear? Something like:

Third-party projects (not supported by the Spark project) exist to add support for Kubernetes and Nomad as cluster managers.

[srowen]

That's a reasonable suggestion, though the K8S integration is intended to come back into Spark soon. Hence doing nothing here is also about the right thing in the near term, even if it's not consistent right now.

[barnardb]

I've updated the PR to leave the Kubernetes links as they are, and just to add a link to the Nomad integration project with wording indicating that it's not supported by the Spark project.

[srowen]

OK by me. You can add to the powered by page if you open a PR vs the spark-website repo

[srowen]

merged to master