[Feat][Format] Format Specification as part of docs

[SemyonSinchenko]

Is your feature request related to a problem? Please describe.
I would like to see an automatically generated format specification from the reference implementation (cpp?);

For example, currently it is not clear which features (nullability, etc.) were added to Cpp implementation and are missing in spark/pyspark.

Describe the solution you'd like
An automatically generated specification from the Cpp code is placed somewhere in docs. With detailed structure of YAML file, folders structure, etc. Especially for YAML files. May it be implemented as placing a generated fake YAML for GraphInfo, EdgeInfo and VertexInfo via Cpp impl? Or a detailed example of such YAML files.

Describe alternatives you've considered
Any other solution could be cool too. I'm not enough familiar with Cpp to say what a canonical way is to do it.

Additional context
I know that there are some examples, but it could be cool to have something in docs too. With more detailed description about what should be inside. And as I understand, this part of the documentation is maintained manually, so it may be outdated.

[acezen]

yes, this's a good advice! I'll see how to generate such doc.

[acezen]

also suggest in issue #352

[acezen]

The auto-generated from cpp is not a good solution I think. Currently the format specification is outdated, I will update it base on our latest design of the format.

The format specification should keep separate from the libraries and the libraries' implementation needs to base on the format design.
So when we discuss to add some new feature to our format, we should first update the format specification, and then update the libraries to support the feature. The process was not run correctly before, I will add the process to the developer guide to ensure we would always develop with the process in the future.

• update the format specification document
• Add correct format design process to developer guide

[lixueclaire]

this part

The auto-generated from cpp is not a good solution I think. Currently the format specification is outdated, I will update it base on our latest design of the format.

The format specification should keep separate from the libraries and the libraries' implementation needs to base on the format design. So when we discuss to add some new feature to our format, we should first update the format specification, and then update the libraries to support the feature. The process was not run correctly before, I will add the process to the developer guide to ensure we would always develop with the process in the future.

• update the format specification document
• Add correct format design process to developer guide

I recommend updating the format page in the documentation with the following entry to reflect the most recent changes:
Last modified Feb XX, 2024: Sync site with format release version x.x.x (commit xxxx)
referencing Parquet.