[Improvement][Doc] Provide an implementation status page to indicate libraries status of format implementation support

[acezen]

…libraries status of format implementation support

Proposed changes

This changes add some tables to indicate the format implementation status.

Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

• I have read the CONTRIBUTING doc
• I have signed the CLA
• Lint and unit tests pass locally with my changes
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)

Further comments

close #352
related #362

[SemyonSinchenko]

Just a small suggestion, related to Spark/PySpark. Because we are going to provide the support of multiple Spark/PySpark versions, I would suggest having the table like, for example:

Graphar Spark Version	Apache Spark versions	Scala version	Java version	Supported GAR format versions
1.0	3.2.x-3.3.x	2.12.10	1.8	0.1-1.0

Because it is important not just know the supported GAR version but also the version of Spark on which the GraphAr Spark will work.

[acezen]

Just a small suggestion, related to Spark/PySpark. Because we are going to provide the support of multiple Spark/PySpark versions, I would suggest having the table like, for example:

Graphar Spark Version Apache Spark versions Scala version Java version Supported GAR format versions
1.0 3.2.x-3.3.x 2.12.10 1.8 0.1-1.0
Because it is important not just know the supported GAR version but also the version of Spark on which the GraphAr Spark will work.

Thanks for the suggestion, Sem. Yes, it's better to provide a table of supported Spark version too. I will add such a page in this Pull Request too.

[acezen]

Just a small suggestion, related to Spark/PySpark. Because we are going to provide the support of multiple Spark/PySpark versions, I would suggest having the table like, for example:
Graphar Spark Version Apache Spark versions Scala version Java version Supported GAR format versions
1.0 3.2.x-3.3.x 2.12.10 1.8 0.1-1.0
Because it is important not just know the supported GAR version but also the version of Spark on which the GraphAr Spark will work.

Thanks for the suggestion, Sem. Yes, it's better to provide a table of supported Spark version too. I will add such a page in this Pull Request too.

I have added tables for libraries to provide version compatibility. @SemyonSinchenko @Thespica Can you help to complement the Java and PySpark part?

[SemyonSinchenko]

@acezen what do you think about publishing this table in releases too?

[acezen]

@acezen what do you think about publishing this table in releases too?

You mean the release notes? I think it's a good place to put the version compatibility table.

[lixueclaire]

hi，@acezen. Could we add some information about supported readers/writers? Also, highlight advanced features like validation and filter pushdown?

[acezen]

I will post the version compatibility table to release note after PR been merged.

[acezen]

hi，@acezen. Could we add some information about supported readers/writers? Also, highlight advanced features like validation and filter pushdown?

The reader/ writers implementation are quite different between C++ and Scala/Python. the C++ library has low level to high level reader/writer, and mainly implement with arrow. The Scala library aims to Spark DataFrame. I'm not sure we can show the status like format implementation.

[acezen]

hi，@acezen. Could we add some information about supported readers/writers? Also, highlight advanced features like validation and filter pushdown?

BTW, I think format implementation are mainly about the meta-info implementation. The reader and writer can put to another document page. We should create a discussion topic to discuss this problem.

[lixueclaire]

hi，@acezen. Could we add some information about supported readers/writers? Also, highlight advanced features like validation and filter pushdown?

The reader/ writers implementation are quite different between C++ and Scala/Python. the C++ library has low level to high level reader/writer, and mainly implement with arrow. The Scala library aims to Spark DataFrame. I'm not sure we can show the status like format implementation.

Maybe a top-level overview of reader/writer support across C++ and Scala/Python, emphasizing the integration with Arrow for C++ and Spark DataFrame for Scala, would be beneficial. We can include a status summary and provide a link to the API documentation for those seeking more in-depth information. Shall we proceed with this approach?

[acezen]

hi，@acezen. Could we add some information about supported readers/writers? Also, highlight advanced features like validation and filter pushdown?

The reader/ writers implementation are quite different between C++ and Scala/Python. the C++ library has low level to high level reader/writer, and mainly implement with arrow. The Scala library aims to Spark DataFrame. I'm not sure we can show the status like format implementation.

Maybe a top-level overview of reader/writer support across C++ and Scala/Python, emphasizing the integration with Arrow for C++ and Spark DataFrame for Scala, would be beneficial. We can include a status summary and provide a link to the API documentation for those seeking more in-depth information. Shall we proceed with this approach?

Sounds reasonable, I can draft a status of reader/writer with this approach.

[acezen]

hi，@acezen. Could we add some information about supported readers/writers? Also, highlight advanced features like validation and filter pushdown?

BTW, I think format implementation are mainly about the meta-info implementation. The reader and writer can put to another document page. We should create a discussion topic to discuss this problem.

hi, @lixueclaire , after add some draft to the status, I'm still thinking it's a little weird that put reader/writer implementation and format implementation together. So I suggest we can add another document to describe the reader/writer implementation. And this change can only contain the format implementation.

[lixueclaire]

hi，@acezen. Could we add some information about supported readers/writers? Also, highlight advanced features like validation and filter pushdown?

BTW, I think format implementation are mainly about the meta-info implementation. The reader and writer can put to another document page. We should create a discussion topic to discuss this problem.

hi, @lixueclaire , after add some draft to the status, I'm still thinking it's a little weird that put reader/writer implementation and format implementation together. So I suggest we can add another document to describe the reader/writer implementation. And this change can only contain the format implementation.

Understood, notice that you put this page under format, I think it is ok. We could include the status of reader/writer implementations in the libraries' documentation later.

[SemyonSinchenko]

GraphAr spark: Java 1.8 and 11, Scala 2.12.x, Hadoop 3, Spark 3.2.2 and 3.3.4
GraphAr PySpark: Java 1.8 and 11, Scala 2.12.x, Hadoop 3, PySpark 3.2.2

(we did not test PySpark against 3.3.4 yet)

[acezen]

GraphAr spark: Java 1.8 and 11, Scala 2.12.x, Hadoop 3, Spark 3.2.2 and 3.3.4 GraphAr PySpark: Java 1.8 and 11, Scala 2.12.x, Hadoop 3, PySpark 3.2.2

(we did not test PySpark against 3.3.4 yet)

Updated, please take a look again.

[SemyonSinchenko]

LGTM