[SPARK-14817][ML][MLLIB][DOC] Made DataFrame-based API primary in MLlib guide

docs/_data/menu-ml.yaml

@@ -1,5 +1,5 @@
-- text: "Overview: estimators, transformers and pipelines"
-  url: ml-guide.html
+- text: Pipelines
+  url: ml-pipeline.html
 - text: Extracting, transforming and selecting features
   url: ml-features.html
 - text: Classification and Regression
@@ -8,5 +8,7 @@
   url: ml-clustering.html
 - text: Collaborative filtering
   url: ml-collaborative-filtering.html
+- text: Model selection and tuning
+  url: ml-tuning.html
 - text: Advanced topics
   url: ml-advanced.html

docs/_includes/nav-left-wrapper-ml.html

@@ -1,8 +1,8 @@
 <div class="left-menu-wrapper">
     <div class="left-menu">
-        <h3><a href="ml-guide.html">spark.ml package</a></h3>
+        <h3><a href="ml-guide.html">MLlib: Main Guide</a></h3>
         {% include nav-left.html nav=include.nav-ml %}
-        <h3><a href="mllib-guide.html">spark.mllib package</a></h3>
+        <h3><a href="mllib-guide.html">MLlib: RDD-based API Guide</a></h3>
         {% include nav-left.html nav=include.nav-mllib %}
     </div>
 </div>

docs/_layouts/global.html

@@ -74,7 +74,7 @@
                                 <li><a href="streaming-programming-guide.html">Spark Streaming</a></li>
                                 <li><a href="sql-programming-guide.html">DataFrames, Datasets and SQL</a></li>
                                 <li><a href="structured-streaming-programming-guide.html">Structured Streaming</a></li>
-                                <li><a href="mllib-guide.html">MLlib (Machine Learning)</a></li>
+                                <li><a href="ml-guide.html">MLlib (Machine Learning)</a></li>
                                 <li><a href="graphx-programming-guide.html">GraphX (Graph Processing)</a></li>
                                 <li><a href="sparkr.html">SparkR (R on Spark)</a></li>
                             </ul>

docs/index.md

@@ -8,7 +8,7 @@ description: Apache Spark SPARK_VERSION_SHORT documentation homepage
 Apache Spark is a fast and general-purpose cluster computing system.
 It provides high-level APIs in Java, Scala, Python and R,
 and an optimized engine that supports general execution graphs.
-It also supports a rich set of higher-level tools including [Spark SQL](sql-programming-guide.html) for SQL and structured data processing, [MLlib](mllib-guide.html) for machine learning, [GraphX](graphx-programming-guide.html) for graph processing, and [Spark Streaming](streaming-programming-guide.html).
+It also supports a rich set of higher-level tools including [Spark SQL](sql-programming-guide.html) for SQL and structured data processing, [MLlib](ml-guide.html) for machine learning, [GraphX](graphx-programming-guide.html) for graph processing, and [Spark Streaming](streaming-programming-guide.html).
 
 # Downloading
 
@@ -87,7 +87,7 @@ options for deployment:
 * Modules built on Spark:
   * [Spark Streaming](streaming-programming-guide.html): processing real-time data streams
   * [Spark SQL, Datasets, and DataFrames](sql-programming-guide.html): support for structured data and relational queries
-  * [MLlib](mllib-guide.html): built-in machine learning library
+  * [MLlib](ml-guide.html): built-in machine learning library
   * [GraphX](graphx-programming-guide.html): Spark's new API for graph processing
 
 **API Docs:**

docs/ml-advanced.md

@@ -1,7 +1,7 @@
 ---
 layout: global
-title: Advanced topics - spark.ml
-displayTitle: Advanced topics - spark.ml
+title: Advanced topics
+displayTitle: Advanced topics
 ---
 
 * Table of contents

docs/ml-ann.md

@@ -1,7 +1,7 @@
 ---
 layout: global
-title: Multilayer perceptron classifier - spark.ml
-displayTitle: Multilayer perceptron classifier - spark.ml
+title: Multilayer perceptron classifier
+displayTitle: Multilayer perceptron classifier
 ---
 
   > This section has been moved into the

docs/ml-classification-regression.md

@@ -1,7 +1,7 @@
 ---
 layout: global
-title: Classification and regression - spark.ml
-displayTitle: Classification and regression - spark.ml
+title: Classification and regression
+displayTitle: Classification and regression
 ---
 
 
@@ -22,37 +22,14 @@ displayTitle: Classification and regression - spark.ml
 \newcommand{\zero}{\mathbf{0}}
 \]`
 
+This page covers algorithms for Classification and Regression.  It also includes sections
+discussing specific classes of algorithms, such as linear methods, trees, and ensembles.
+
 **Table of Contents**
 
 * This will become a table of contents (this text will be scraped).
 {:toc}
 
-In `spark.ml`, we implement popular linear methods such as logistic
-regression and linear least squares with $L_1$ or $L_2$ regularization.
-Refer to [the linear methods in mllib](mllib-linear-methods.html) for
-details about implementation and tuning.  We also include a DataFrame API for [Elastic
-net](http://en.wikipedia.org/wiki/Elastic_net_regularization), a hybrid
-of $L_1$ and $L_2$ regularization proposed in [Zou et al, Regularization
-and variable selection via the elastic
-net](http://users.stat.umn.edu/~zouxx019/Papers/elasticnet.pdf).
-Mathematically, it is defined as a convex combination of the $L_1$ and
-the $L_2$ regularization terms:
-`\[
-\alpha \left( \lambda \|\wv\|_1 \right) + (1-\alpha) \left( \frac{\lambda}{2}\|\wv\|_2^2 \right) , \alpha \in [0, 1], \lambda \geq 0
-\]`
-By setting $\alpha$ properly, elastic net contains both $L_1$ and $L_2$
-regularization as special cases. For example, if a [linear
-regression](https://en.wikipedia.org/wiki/Linear_regression) model is
-trained with the elastic net parameter $\alpha$ set to $1$, it is
-equivalent to a
-[Lasso](http://en.wikipedia.org/wiki/Least_squares#Lasso_method) model.
-On the other hand, if $\alpha$ is set to $0$, the trained model reduces
-to a [ridge
-regression](http://en.wikipedia.org/wiki/Tikhonov_regularization) model.
-We implement Pipelines API for both linear regression and logistic
-regression with elastic net regularization.
-
-
 # Classification
 
 ## Logistic regression
@@ -760,7 +737,34 @@ Refer to the [`IsotonicRegression` Python docs](api/python/pyspark.ml.html#pyspa
 </div>
 </div>
 
+# Linear methods
+
+We implement popular linear methods such as logistic
+regression and linear least squares with $L_1$ or $L_2$ regularization.
+Refer to [the linear methods guide for the RDD-based API](mllib-linear-methods.html) for
+details about implementation and tuning; this information is still relevant.
 
+We also include a DataFrame API for [Elastic
+net](http://en.wikipedia.org/wiki/Elastic_net_regularization), a hybrid
+of $L_1$ and $L_2$ regularization proposed in [Zou et al, Regularization
+and variable selection via the elastic
+net](http://users.stat.umn.edu/~zouxx019/Papers/elasticnet.pdf).
+Mathematically, it is defined as a convex combination of the $L_1$ and
+the $L_2$ regularization terms:
+`\[
+\alpha \left( \lambda \|\wv\|_1 \right) + (1-\alpha) \left( \frac{\lambda}{2}\|\wv\|_2^2 \right) , \alpha \in [0, 1], \lambda \geq 0
+\]`
+By setting $\alpha$ properly, elastic net contains both $L_1$ and $L_2$
+regularization as special cases. For example, if a [linear
+regression](https://en.wikipedia.org/wiki/Linear_regression) model is
+trained with the elastic net parameter $\alpha$ set to $1$, it is
+equivalent to a
+[Lasso](http://en.wikipedia.org/wiki/Least_squares#Lasso_method) model.
+On the other hand, if $\alpha$ is set to $0$, the trained model reduces
+to a [ridge
+regression](http://en.wikipedia.org/wiki/Tikhonov_regularization) model.
+We implement Pipelines API for both linear regression and logistic
+regression with elastic net regularization.
 
 # Decision trees
 

docs/ml-clustering.md

@@ -1,10 +1,12 @@
 ---
 layout: global
-title: Clustering - spark.ml
-displayTitle: Clustering - spark.ml
+title: Clustering
+displayTitle: Clustering
 ---
 
-In this section, we introduce the pipeline API for [clustering in mllib](mllib-clustering.html).
+This page describes clustering algorithms in MLlib.
+The [guide for clustering in the RDD-based API](mllib-clustering.html) also has relevant information
+about these algorithms.
 
 **Table of Contents**
 

docs/ml-collaborative-filtering.md

@@ -1,7 +1,7 @@
 ---
 layout: global
-title: Collaborative Filtering - spark.ml
-displayTitle: Collaborative Filtering - spark.ml
+title: Collaborative Filtering
+displayTitle: Collaborative Filtering
 ---
 
 * Table of contents

docs/ml-decision-tree.md

@@ -1,7 +1,7 @@
 ---
 layout: global
-title: Decision trees - spark.ml
-displayTitle: Decision trees - spark.ml
+title: Decision trees
+displayTitle: Decision trees
 ---
 
   > This section has been moved into the

docs/ml-ensembles.md

@@ -1,7 +1,7 @@
 ---
 layout: global
-title: Tree ensemble methods - spark.ml
-displayTitle: Tree ensemble methods - spark.ml
+title: Tree ensemble methods
+displayTitle: Tree ensemble methods
 ---
 
   > This section has been moved into the

docs/ml-features.md

@@ -1,7 +1,7 @@
 ---
 layout: global
-title: Extracting, transforming and selecting features - spark.ml
-displayTitle: Extracting, transforming and selecting features - spark.ml
+title: Extracting, transforming and selecting features
+displayTitle: Extracting, transforming and selecting features
 ---
 
 This section covers algorithms for working with features, roughly divided into these groups: