[DOCS]: consolidate doc site content simplify navbar #5962

alamb · 2023-04-11T11:24:12Z

Which issue does this PR close?

Closes #5935
Related to #1814 and #5501

Rationale for this change

The main page / index of the https://arrow.apache.org/datafusion/ site is somewhat disorganized, redundant and has so many entries it is causing issues such as #5935

Here is a screenshot of the current site:

What changes are included in this PR?

Consolidate some top level pages into lower level pages ("improve the site navigation") to reduce the size of the initial index and organize the content better.
Various small improvements

Are these changes tested?

I rendered the site locally and it looks better to me:

Are there any user-facing changes?

alamb · 2023-04-11T11:26:48Z

docs/source/index.rst

@@ -50,22 +49,17 @@ community.

   user-guide/introduction
   user-guide/example-usage
-   user-guide/users


I consolidated the content of these pages into other pages

alamb · 2023-04-11T11:27:04Z

docs/source/index.rst

   user-guide/cli
   user-guide/dataframe
   user-guide/expressions
   user-guide/sql/index
   user-guide/configs
   user-guide/faq
-   Rust Crate Documentation <https://docs.rs/crate/datafusion/>


This was redundant with the crates.io link above

alamb · 2023-04-11T11:27:40Z

docs/source/index.rst


 .. _toc.contributor-guide:

 .. toctree::
-   :maxdepth: 2


This stops listing H2 headings ( ##) on the main table of contents

Jefffrey

Hope you don't mind but took the opportunity to review the docs and point out a few parts that are outdated/could be improved

Jefffrey · 2023-04-11T12:20:41Z

docs/source/user-guide/example-usage.md

+
+```toml
+[dependencies]
+datafusion = "11.0"


bump to latest here? (ditto for anywhere else version is mentioned)

Jefffrey · 2023-04-11T12:22:33Z

docs/source/user-guide/example-usage.md

+
+## Create a main function
+
+Update the main.rs file with your first datafusion application based on [Example usage](https://arrow.apache.org/datafusion/user-guide/example-usage.html)


is this self link to same page, this page?

Yes, that is a good a catch. This whole page has some non trivial redundancy. I will try and fix it up

Jefffrey · 2023-04-11T12:23:48Z

docs/source/user-guide/example-usage.md

+```rust
+use datafusion::prelude::*;
+
+#[tokio::main]
+async fn main() -> datafusion::error::Result<()> {
+  // register the table
+  let ctx = SessionContext::new();
+  ctx.register_csv("test", "<PATH_TO_YOUR_CSV_FILE>", CsvReadOptions::new()).await?;
+
+  // create a plan to run a SQL query
+  let df = ctx.sql("SELECT * FROM test").await?;
+
+  // execute and print results
+  df.show().await?;
+  Ok(())
+}
+```


example feels kinda redundant compared with example code in above sections

I agree -- removed

Jefffrey · 2023-04-11T12:26:04Z

docs/source/user-guide/faq.md

+Here is a comparison with similar projects that may help understand
+when DataFusion might be be suitable and unsuitable for your needs:
+
+- [DuckDB](http://www.duckdb.org) is an open source, in process analytic database.


change to https link?

in c187863

Jefffrey · 2023-04-11T12:26:41Z

docs/source/user-guide/faq.md

+- [Polars](http://pola.rs): Polars is one of the fastest DataFrame
+  libraries at the time of writing. Like DataFusion, it is also
+  written in Rust and uses the Apache Arrow memory model, but unlike
+  DataFusion it does not provide SQL nor as many extension points.


change to https url

also i think polars might support sql now, according to their doc: https://pola-rs.github.io/polars-book/user-guide/sql.html

Yes I think you are correct.

Jefffrey · 2023-04-11T12:27:56Z

docs/source/user-guide/faq.md

+  written in Rust and uses the Apache Arrow memory model, but unlike
+  DataFusion it does not provide SQL nor as many extension points.
+
+- [Facebook Velox](https://engineering.fb.com/2022/08/31/open-source/velox/)


could switch to github link: https://github.com/facebookincubator/velox since this link seems dead

waynexia

Thanks for bringing up this topic. I also left some unrelated comment 🙇‍♂️

waynexia · 2023-04-12T07:42:33Z

docs/source/contributor-guide/architecture.md

-[Crate Documentation], to keep it as close to the source as
-possible.
+[crates.io documentation], to keep it as close to the source as
+possible. You can find the most up to date version in the [source code].


What do you think about hosting the latest document generated from the source code on github pages (or other static page hoster)? Like greptimedb.rs which is generated from https://github.com/GreptimeTeam/greptimedb/deployments/activity_log?environment=github-pages

I think it is a great idea -- thank you @waynexia

I actually think if we could build those API docs as part of the https://github.com/apache/arrow-datafusion/blob/main/docs build, they would "automatically" get hosted on https://arrow.apache.org/datafusion/

https://arrow.apache.org/datafusion/ is published via some ASF mechanism that is similar to github pages

Specifically, this workflow

https://github.com/apache/arrow-datafusion/blob/388f9ec3e7f7c09dac56ee0fe074ca97a6af9d44/.github/workflows/docs.yaml#L12-L64

pushes to the https://github.com/apache/arrow-datafusion/tree/asf-site branch which then gets hosted via this magic yaml:

https://github.com/apache/arrow-datafusion/blob/388f9ec3e7f7c09dac56ee0fe074ca97a6af9d44/.asf.yaml#L48-L52

Filed #5981

docs/source/user-guide/faq.md

waynexia · 2023-04-12T07:53:45Z

docs/source/user-guide/example-usage.md

+
+```toml
+[dependencies]
+datafusion = { version = "11.0" , features = ["simd"]}


This is also outdated. Wondering if there are someway to render code / file from github? So we needn't update this file every time but rather render our example codes. I find this but it looks only works inside github.

Maybe we could update the script here to automatically clean it up: https://github.com/apache/arrow-datafusion/blob/main/dev/update_datafusion_versions.py

Sounds good, I filed #5983

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>

alamb · 2023-04-12T19:56:31Z

Hope you don't mind but took the opportunity to review the docs and point out a few parts that are outdated/could be improved

Not at all - thank you @Jefffrey and thank you @waynexia

alamb · 2023-04-12T20:22:38Z

Ok, I think this is better than what is on main so I will merge it in. We clearly have a ways to go to have wonderful docs

* [DOCS]: consolidate doc site content simplify navbar * prettier * Update docs/source/user-guide/faq.md Co-authored-by: Ruihang Xia <waynestxia@gmail.com> * Update versions to latest * remove reundant example * update duckdb link and polars description * update velox link * prettier --------- Co-authored-by: Ruihang Xia <waynestxia@gmail.com>

alamb added 2 commits April 11, 2023 07:18

[DOCS]: consolidate doc site content simplify navbar

94457bf

prettier

75d7d9e

alamb mentioned this pull request Apr 11, 2023

doc site sidebar icon is glitched/bugged #5935

Closed

alamb requested a review from andygrove April 11, 2023 11:26

alamb commented Apr 11, 2023

View reviewed changes

Jefffrey reviewed Apr 11, 2023

View reviewed changes

waynexia reviewed Apr 12, 2023

View reviewed changes

Update docs/source/user-guide/faq.md

3dba7ac

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>

alamb added 6 commits April 12, 2023 15:57

Merge remote-tracking branch 'apache/main' into alamb/fix_doc_sidebar

fbc699c

Update versions to latest

bb98e7b

remove reundant example

6f6b729

update duckdb link and polars description

c187863

update velox link

898f850

prettier

3ac0f52

alamb merged commit 4c7833e into apache:main Apr 12, 2023

alamb deleted the alamb/fix_doc_sidebar branch April 12, 2023 20:22

alamb mentioned this pull request Apr 12, 2023

Include the latest rustdocs (e.g. cargo doc output on https://arrow.apache.org/datafusion/) #5981

Open

waynexia mentioned this pull request Apr 13, 2023

Include example codes in version update script #5983

Open

alamb mentioned this pull request Apr 13, 2023

minor: fix doctest that runs accidentally #5989

Merged

waynexia mentioned this pull request Apr 15, 2023

CI Fail on cargo doctest #5986

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DOCS]: consolidate doc site content simplify navbar #5962

[DOCS]: consolidate doc site content simplify navbar #5962

alamb commented Apr 11, 2023 •

edited

Loading

alamb Apr 11, 2023

alamb Apr 11, 2023

alamb Apr 11, 2023

Jefffrey left a comment

Jefffrey Apr 11, 2023

Jefffrey Apr 11, 2023

alamb Apr 12, 2023

Jefffrey Apr 11, 2023

alamb Apr 12, 2023

Jefffrey Apr 11, 2023

alamb Apr 12, 2023

Jefffrey Apr 11, 2023

alamb Apr 12, 2023

Jefffrey Apr 11, 2023

waynexia left a comment

waynexia Apr 12, 2023

alamb Apr 12, 2023

alamb Apr 12, 2023

waynexia Apr 12, 2023

alamb Apr 12, 2023

waynexia Apr 13, 2023

alamb commented Apr 12, 2023

alamb commented Apr 12, 2023


		## Create a main function

		Update the main.rs file with your first datafusion application based on [Example usage](https://arrow.apache.org/datafusion/user-guide/example-usage.html)

[DOCS]: consolidate doc site content simplify navbar #5962

[DOCS]: consolidate doc site content simplify navbar #5962

Conversation

alamb commented Apr 11, 2023 • edited Loading

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Jefffrey left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

waynexia left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alamb commented Apr 12, 2023

alamb commented Apr 12, 2023

alamb commented Apr 11, 2023 •

edited

Loading