Skip to content

[SPARK-18793] [SPARK-18794] [R] add spark.randomForest/spark.gbt to vignettes #16264

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
mengxr wants to merge 4 commits into from
Closed

[SPARK-18793] [SPARK-18794] [R] add spark.randomForest/spark.gbt to vignettes #16264

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
292745b
mention spark.randomForest in vignettes
mengxr Dec 13, 2016
fb1933d
add spark.gbt as well
mengxr Dec 13, 2016
be9c846
update params so the output is shorter
mengxr Dec 13, 2016
b3bf19f
address comments
mengxr Dec 13, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions R/pkg/vignettes/sparkr-vignettes.Rmd
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -449,6 +449,10 @@ SparkR supports the following machine learning models and algorithms.

* Generalized Linear Model (GLM)

* Random Forest

* Gradient-Boosted Trees (GBT)

* Naive Bayes Model

* $k$-means Clustering
Expand Down Expand Up @@ -526,6 +530,34 @@ gaussianFitted <- predict(gaussianGLM, carsDF)
head(select(gaussianFitted, "model", "prediction", "mpg", "wt", "hp"))
```

#### Random Forest

`spark.randomForest` fits a [random forest](https://en.wikipedia.org/wiki/Random_forest) classification or regression model on a `SparkDataFrame`.
Users can call `summary` to get a summary of the fitted model, `predict` to make predictions, and `write.ml`/`read.ml` to save/load fitted models.

In the following example, we use the `longley` dataset to train a random forest and make predictions:

```{r, warning=FALSE}
df <- createDataFrame(longley)
rfModel <- spark.randomForest(df, Employed ~ ., type = "regression", maxDepth = 2, numTrees = 2)
Copy link
Member

@felixcheung felixcheung Dec 13, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you could also do this to limit output

ops <- options()
options(max.print=40)

there is another example in the vignettes

Copy link
Contributor Author

@mengxr mengxr Dec 13, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current length of the output seems fine (<40 lines).

summary(rfModel)
predictions <- predict(rfModel, df)
```

#### Gradient-Boosted Trees

`spark.gbt` fits a [gradient-boosted tree](https://en.wikipedia.org/wiki/Gradient_boosting) classification or regression model on a `SparkDataFrame`.
Users can call `summary` to get a summary of the fitted model, `predict` to make predictions, and `write.ml`/`read.ml` to save/load fitted models.

Similar to the random forest example above, we use the `longley` dataset to train a gradient-boosted tree and make predictions:

```{r, warning=FALSE}
df <- createDataFrame(longley)
gbtModel <- spark.gbt(df, Employed ~ ., type = "regression", maxDepth = 2, maxIter = 2)
summary(gbtModel)
predictions <- predict(gbtModel, df)
```

#### Naive Bayes Model

Naive Bayes model assumes independence among the features. `spark.naiveBayes` fits a [Bernoulli naive Bayes model](https://en.wikipedia.org/wiki/Naive_Bayes_classifier#Bernoulli_naive_Bayes) against a SparkDataFrame. The data should be all categorical. These models are often used for document classification.
Expand Down