GH-38837: [Format] Add the specification for statistics schema #45058
Conversation
|
@github-actions crossbow submit preview-docs |
|
|
github-actions
bot
added
Component: Documentation
awaiting committer review
kou
changed the title
|
Revision: 5308f9f Submitted crossbow builds: ursacomputing/crossbow @ actions-a2d5a054aa
|
| can access to proper field by type code not name. So we can use | ||
| any valid name for fields. | ||
|
|
||
| TODO: Should we standardize field names? |
There was a problem hiding this comment.
I don't see any reason to standardize the names, but a reason I could see to use explicit type IDs for at least a few commonly used statistic types would be to ensure that an ArrowArray (or standalone RecordBatch message) could be interpreted without a ArrowSchema. (Completely optional!)
There was a problem hiding this comment.
Related discussion: https://github.com/apache/arrow/pull/43553/files#r1871755575
| - Nullable | ||
| - Notes | ||
| * - key | ||
| - ``dictionary<indices: int32, dictionary: utf8>`` |
There was a problem hiding this comment.
The only reason I can see why this would be problematic is that the statistics values would require more than one IPC message to represent. (Completely optional: this may not be an important consideration!)
There was a problem hiding this comment.
Ah, I didn't notice the point. Thanks.
The current proposed specification doesn't focus on any transports/protocols/APIs/.... So this may not be a problem. If this representation doesn't match for our IPC formats, users just don't use this.
| - The maximum size in bytes of a row in the target | ||
| column. (exact) | ||
| * - ``ARROW:max_byte_width:approximate`` | ||
| - ``float64``: TODO: Should we use ``int64`` instead? |
There was a problem hiding this comment.
The float64ness makes sense to me here because the calculation providing this approximate value almost certainly returns a non-exact value (i.e., not an integer, even though the exact value is definitely an integer).
There was a problem hiding this comment.
Related discussion: https://github.com/apache/arrow/pull/43553/files#r1871759234
github-actions
bot
added
awaiting changes
|
Thanks for sharing your opinions! |
github-actions
bot
added
awaiting changes
Rationale for this change
Statistics are useful for fast query processing. Many query engines
use statistics to optimize their query plan.
Apache Arrow format doesn't have statistics but other formats that can
be read as Apache Arrow data may have statistics. For example, Apache
Parquet C++ can read Apache Parquet file as Apache Arrow data and
Apache Parquet file may have statistics.
One of the Apache Arrow C streaming interface use cases is the following:
Arrow C data interface
If module A can pass the statistics associated with the Apache Parquet
file to module B, module B can use the statistics to optimize its
query plan.
What changes are included in this PR?
We standardize how to represent statistics as an Apache Arrow array
for easy to exchange.
We don't standardize how to pass the statistics array. You can use any
interface for it. For example, you can us ethe Apache Arrow C data interface.
Are these changes tested?
Yes.
Are there any user-facing changes?
Yes.