[SPARK-8345] [ML] Add an SQL node as a feature transformer

[yanboliang]

Implements the transforms which are defined by SQL statement.
Currently we only support SQL syntax like 'SELECT ... FROM THIS'
where 'THIS' represents the underlying table of the input dataset.

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[SparkQA]

Test build #37623 has started for PR 7465 at commit 51eb9e7.

[SparkQA]

Test build #37623 has finished for PR 7465 at commit 51eb9e7.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class SQLTransformer (override val uid: String) extends Transformer

[AmplabJenkins]

Merged build finished. Test PASSed.

[mengxr]

I see the issue but I don't think this is the right solution. See my comments below.

[liancheng]

@mengxr I guess what we need here is essentially a wrapper helper function which wraps a DataFrame => DataFrame function as a transformer, and a SQL statement is just a (questionably) more convenient way to express this function. One of the benefit of DataFrame DSL over SQL is that you don't need a temporary table name.

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[SparkQA]

Test build #37827 has started for PR 7465 at commit 0d4bb15.

[SparkQA]

Test build #37827 has finished for PR 7465 at commit 0d4bb15.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class SQLTransformer (override val uid: String) extends Transformer

[AmplabJenkins]

Merged build finished. Test PASSed.

[yanboliang]

@mengxr

[mengxr]

I think it is okay to return the DataFrame from sqlContext.sql directly. User should use * if they want to keep existing columns.

[yanboliang]

This will have different behavior with other transformers in ml.feature. Other transformers will return the DataFrame which is composed of original DataFrame and transformed DataFrame. But here if user did not use *, he will not keep existing columns in the output DataFrame.

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[SparkQA]

Test build #40267 has started for PR 7465 at commit b403fcb.

[SparkQA]

Test build #40267 has finished for PR 7465 at commit b403fcb.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class SQLTransformer (override val uid: String) extends Transformer

[AmplabJenkins]

Merged build finished. Test PASSed.

[mengxr]

LGTM. Merged into master. I think it is okay if the output columns do not contain all input columns. It is not a requirement for transformers in the pipeline API. Thanks for working on this!