[SPARK-35159][SQL][DOCS] Extract hive format doc #32264

AngersZhuuuu · 2021-04-21T03:09:43Z

What changes were proposed in this pull request?

Extract common doc about hive format for sql-ref-syntax-ddl-create-table-hiveformat.md and sql-ref-syntax-qry-select-transform.md to refer.

Why are the changes needed?

Improve doc

Does this PR introduce any user-facing change?

No

How was this patch tested?

Not need

AngersZhuuuu · 2021-04-21T03:15:27Z

I am not sure if we need to put this page in which menu page

SparkQA · 2021-04-21T03:22:39Z

Test build #137710 has finished for PR 32264 at commit df246ea.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
SERDE serde_class [ WITH SERDEPROPERTIES (k1=v1, k2=v2, ... ) ]

SparkQA · 2021-04-21T03:25:36Z

Test build #137711 has finished for PR 32264 at commit d98f825.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-04-21T04:00:48Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42239/

SparkQA · 2021-04-21T04:00:49Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42239/

SparkQA · 2021-04-21T04:05:05Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42238/

SparkQA · 2021-04-21T04:05:06Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42238/

cloud-fan · 2021-04-21T07:53:38Z

docs/sql-ref-syntax-hive-format.md

+
+### Description
+
+Spark support Hive format in `CREATE TABLE` clause and `TRANSFORM` clause, Hive format support 


Spark supports Hive format in `CREATE TABLE` clause and `TRANSFORM` clause, to specify serde or text delimeter.

cloud-fan · 2021-04-21T07:54:45Z

docs/sql-ref-syntax-hive-format.md

+
+* **row_format**    
+
+    Use the `SERDE` clause to specify a custom SerDe for one table or processing inputs and outputs data. Otherwise, use the `DELIMITED` clause to use the native SerDe and specify the delimiter, escape character, null character and so on.


shall we put it in Description at the beginning?

cloud-fan · 2021-04-21T07:55:32Z

docs/sql-ref-syntax-hive-format.md

+
+    Use the `SERDE` clause to specify a custom SerDe for one table or processing inputs and outputs data. Otherwise, use the `DELIMITED` clause to use the native SerDe and specify the delimiter, escape character, null character and so on.
+
+* **SERDE**


we can merge this with the next one. SERDE serde_class

cloud-fan · 2021-04-21T07:57:05Z

not sure if we need to put this page in which menu page

We don't need to put it in the menu page.

SparkQA · 2021-04-21T10:32:25Z

Test build #137732 has finished for PR 32264 at commit 203f544.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-04-21T11:19:06Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42259/

SparkQA · 2021-04-21T11:19:07Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42259/

dongjoon-hyun · 2021-04-22T03:03:48Z

docs/sql-ref-syntax-hive-format.md

+---
+layout: global
+title: Data Retrieval
+displayTitle: Data Retrieval


~~In this case, shall we add a reference to this sql-ref-syntax-hive-format.md into sql-ref-syntax-qry.md?~~

Oh, got it. I saw @cloud-fan 's comment, We don't need to put it in the menu page. Nvm.

Yea, refer this in other menu doc is so strange and it's refer in two syntax doc with different type.

docs/sql-ref-syntax-hive-format.md

cloud-fan · 2021-04-22T08:06:12Z

docs/sql-ref-syntax-hive-format.md

+Spark supports Hive format in `CREATE TABLE` clause and `TRANSFORM` clause,
+to specify serde or text delimeter. In `row_format`, uses the `SERDE` clause to specify a custom SerDe
+for one table or processing inputs and outputs data. Otherwise, use the `DELIMITED` clause
+to use the native SerDe and specify the delimiter, escape character, null character and so on.


how about this

There are two ways to specify the `row_format`: 1. Use the `SERDE` clause to specify a custom SerDe class 2. Use the `DELIMITED` clause to specify the delimiter ... and so on for the native text Serde.

Year, more clear

docs/sql-ref-syntax-hive-format.md

cloud-fan · 2021-04-22T09:01:08Z

@maropu do you want to take a look?

SparkQA · 2021-04-22T09:28:44Z

Test build #137797 has finished for PR 32264 at commit 5fe64b5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-04-22T10:24:07Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42326/

SparkQA · 2021-04-22T10:24:09Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42326/

docs/sql-ref-syntax-hive-format.md

maropu · 2021-04-22T14:53:18Z

docs/sql-ref-syntax-ddl-create-table-hiveformat.md

+* **row_format** 

-    Used for escape mechanism.
+    All descriptions about syntax in `row_format` can refer to [HIVE FORMAT](sql-ref-syntax-hive-format.html)


How about Specifies the row format for input and output. See [HIVE ROW FORMAT](...) for more syntax details.?

docs/sql-ref-syntax-qry-select-transform.md

docs/sql-ref-syntax-hive-format.md

maropu · 2021-04-22T15:03:04Z

Could you add the screenshot of the new page in the PR description?

maropu · 2021-04-22T15:05:29Z

NOTE: I'm planning to backport this PR and #31010 into branch-3.1/3.0 because I think these document pages are useful for users.

AngersZhuuuu · 2021-04-23T00:52:10Z

Could you add the screenshot of the new page in the PR description?

DOne

SparkQA · 2021-04-23T01:24:03Z

Test build #137837 has finished for PR 32264 at commit 9bfa1cf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-04-23T01:55:55Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42367/

SparkQA · 2021-04-23T02:00:31Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42367/

cloud-fan · 2021-04-23T05:47:46Z

thanks, merging to master!

cloud-fan · 2021-04-23T05:48:22Z

thanks, merging to master!

cloud-fan · 2021-04-23T05:48:56Z

@maropu shall we have a single backport PR or two?

maropu · 2021-04-23T06:00:57Z

They have different jira tickets, so I think its better to backport them separately. Could you? @AngersZhuuuu

maropu · 2021-04-23T06:01:46Z

Anyway, late lgtm. Thank you, @AngersZhuuuu

maropu · 2021-04-26T07:56:12Z

They have different jira tickets, so I think its better to backport them separately. Could you? @AngersZhuuuu

ping

AngersZhuuuu · 2021-04-26T08:01:14Z

They have different jira tickets, so I think its better to backport them separately. Could you? @AngersZhuuuu

ping

Hmmm, have conflict? need me to create backport PR?

maropu · 2021-04-26T08:04:00Z

yea, yes. I couldn't cherry-pick them into the previous branches.

AngersZhuuuu · 2021-04-26T08:34:40Z

yea, yes. I couldn't cherry-pick them into the previous branches.

Ok, ping you later when PR is ready

### What changes were proposed in this pull request? Extract common doc about hive format for `sql-ref-syntax-ddl-create-table-hiveformat.md` and `sql-ref-syntax-qry-select-transform.md` to refer. ![image](https://user-images.githubusercontent.com/46485123/115802193-04641800-a411-11eb-827d-d92544881842.png) ### Why are the changes needed? Improve doc ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Not need Closes apache#32264 from AngersZhuuuu/SPARK-35159. Authored-by: Angerszhuuuu <angers.zhu@gmail.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>

[SPARK-35159][SQL][DOCS] Extract hive format doc

df246ea

github-actions bot added the DOCS label Apr 21, 2021

AngersZhuuuu added 3 commits April 21, 2021 11:10

Update sql-ref-syntax-ddl-create-table-hiveformat.md

5d53976

Update sql-ref-syntax-ddl-create-table-hiveformat.md

a49f1e5

Update sql-ref-syntax-qry-select-transform.md

d98f825

AngersZhuuuu mentioned this pull request Apr 21, 2021

[SPARK-33976][SQL][DOCS] Add a SQL doc page for a TRANSFORM clause #31010

Closed

cloud-fan reviewed Apr 21, 2021

View reviewed changes

Update sql-ref-syntax-hive-format.md

203f544

dongjoon-hyun reviewed Apr 22, 2021

View reviewed changes

cloud-fan reviewed Apr 22, 2021

View reviewed changes

docs/sql-ref-syntax-hive-format.md Outdated Show resolved Hide resolved

cloud-fan reviewed Apr 22, 2021

View reviewed changes

docs/sql-ref-syntax-hive-format.md Outdated Show resolved Hide resolved

cloud-fan reviewed Apr 22, 2021

View reviewed changes

docs/sql-ref-syntax-hive-format.md Outdated Show resolved Hide resolved

Update sql-ref-syntax-hive-format.md

5fe64b5

cloud-fan approved these changes Apr 22, 2021

View reviewed changes

maropu reviewed Apr 22, 2021

View reviewed changes

AngersZhuuuu added 2 commits April 23, 2021 08:44

follow comment

be29ae8

Update sql-ref-syntax-hive-format.md

9bfa1cf

cloud-fan closed this in 20d68dc Apr 23, 2021


		### Description

		Spark support Hive format in `CREATE TABLE` clause and `TRANSFORM` clause, Hive format support


		* row_format

		Use the `SERDE` clause to specify a custom SerDe for one table or processing inputs and outputs data. Otherwise, use the `DELIMITED` clause to use the native SerDe and specify the delimiter, escape character, null character and so on.


		Use the `SERDE` clause to specify a custom SerDe for one table or processing inputs and outputs data. Otherwise, use the `DELIMITED` clause to use the native SerDe and specify the delimiter, escape character, null character and so on.

		* SERDE

[SPARK-35159][SQL][DOCS] Extract hive format doc #32264

[SPARK-35159][SQL][DOCS] Extract hive format doc #32264

Uh oh!

Conversation

AngersZhuuuu commented Apr 21, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

AngersZhuuuu commented Apr 21, 2021

Uh oh!

SparkQA commented Apr 21, 2021

Uh oh!

SparkQA commented Apr 21, 2021

Uh oh!

SparkQA commented Apr 21, 2021

Uh oh!

SparkQA commented Apr 21, 2021

Uh oh!

SparkQA commented Apr 21, 2021

Uh oh!

SparkQA commented Apr 21, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cloud-fan Apr 21, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cloud-fan commented Apr 21, 2021

Uh oh!

SparkQA commented Apr 21, 2021

Uh oh!

SparkQA commented Apr 21, 2021

Uh oh!

SparkQA commented Apr 21, 2021

Uh oh!

dongjoon-hyun Apr 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

cloud-fan commented Apr 22, 2021

Uh oh!

SparkQA commented Apr 22, 2021

Uh oh!

SparkQA commented Apr 22, 2021

Uh oh!

SparkQA commented Apr 22, 2021

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

maropu commented Apr 22, 2021

Uh oh!

maropu commented Apr 22, 2021

Uh oh!

AngersZhuuuu commented Apr 23, 2021

AngersZhuuuu commented Apr 21, 2021 •

edited

Loading

cloud-fan Apr 21, 2021 •

edited

Loading

dongjoon-hyun Apr 22, 2021 •

edited

Loading

maropu commented Apr 23, 2021 •

edited

Loading