GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface #43553

kou · 2024-08-05T09:28:14Z

Rationale for this change

Statistics are useful for fast query processing. Many query engines
use statistics to optimize their query plan.

Apache Arrow format doesn't have statistics but other formats that can
be read as Apache Arrow data may have statistics. For example, Apache
Parquet C++ can read Apache Parquet file as Apache Arrow data and
Apache Parquet file may have statistics.

One of the Arrow C data interface use cases is the following:

Module A reads Apache Parquet file as Apache Arrow data
Module A passes the read Apache Arrow data to module B through the
Arrow C data interface
Module B processes the passed Apache Arrow data

If module A can pass the statistics associated with the Apache Parquet
file to module B through the Arrow C data interface, module B can use
the statistics to optimize its query plan.

What changes are included in this PR?

Add the specification to pass statistics through the Arrow C data interface based on the discussion on the dev@ mailing list: https://lists.apache.org/thread/z0jz2bnv61j7c6lbk7lympdrs49f69cx

Are these changes tested?

Yes.

Are there any user-facing changes?

Yes.

GitHub Issue: [Format] Passing column statistics through Arrow C data interface #38837

kou · 2024-08-05T09:28:37Z

@github-actions crossbow submit preview-docs

github-actions · 2024-08-05T09:28:42Z

⚠️ GitHub issue #38837 has been automatically assigned in GitHub to PR creator.

kou · 2024-08-05T09:29:54Z

I'm not a native English speaker. Wording suggestions are very welcome.

I'll add examples after I implement a convenient API to C++.

github-actions · 2024-08-05T09:30:52Z

Revision: 22336f4

Submitted crossbow builds: ursacomputing/crossbow @ actions-28c2a45b3d

Task	Status
preview-docs

docs/source/format/CDataInterfaceStatistics.rst

ianmcook · 2024-08-05T16:29:13Z

docs/source/format/CDataInterfaceStatistics.rst

+* Provide a common way to pass statistics that can be used for
+  other interfaces such Arrow Flight too.


What about the Arrow IPC format? Can you add a sentence here that explains why we do not recommend using this to pass statistics over Arrow IPC?

Good point. This may fit the Arrow IPC format. (A producer sends data and statistics as 2 separated the Arrow IPC format data.)

But the Arrow IPC format can use more approaches. For example, the Arrow IPC format can have metadata for each record batch data: https://arrow.apache.org/docs/format/Columnar.html#encapsulated-message-format (The Arrow C data can't have metadata for ArrowArray.)
The Arrow IPC format can be used with other mechanisms such as Arrow Flight and ADBC.

So this may not be the best approach for the Arrow IPC format. We should discuss this use case with the Arrow IPC format separately.

I'll add something to here.

ianmcook · 2024-08-05T16:50:51Z

docs/source/format/CDataInterfaceStatistics.rst

+For example, ADBC has the statistics related APIs. This specification
+doesn't replace them.


Maybe link to https://arrow.apache.org/adbc/current/format/specification.html#statistics

Thanks. I should have done it.

ianmcook · 2024-08-05T16:53:42Z

@pdet please take a look and add comments if you have any, thanks!

pdet · 2024-08-06T08:11:27Z

The format and contents LGTM! I was just slightly confused for one second that the second mapping is the value items in the first mapping.

@Tmonster I think the proposed Statistics keys already cover the table statistics we use/produce. Could you double-check?

docs/source/format/CDataInterfaceStatistics.rst

kou

@ianmcook Thanks for your suggestions! I've merged all of them!

kou · 2024-08-07T02:25:17Z

docs/source/format/CDataInterfaceStatistics.rst

+* Provide a common way to pass statistics that can be used for
+  other interfaces such Arrow Flight too.


Good point. This may fit the Arrow IPC format. (A producer sends data and statistics as 2 separated the Arrow IPC format data.)

But the Arrow IPC format can use more approaches. For example, the Arrow IPC format can have metadata for each record batch data: https://arrow.apache.org/docs/format/Columnar.html#encapsulated-message-format (The Arrow C data can't have metadata for ArrowArray.)
The Arrow IPC format can be used with other mechanisms such as Arrow Flight and ADBC.

So this may not be the best approach for the Arrow IPC format. We should discuss this use case with the Arrow IPC format separately.

I'll add something to here.

kou · 2024-08-07T02:25:57Z

docs/source/format/CDataInterfaceStatistics.rst

+For example, ADBC has the statistics related APIs. This specification
+doesn't replace them.


Thanks. I should have done it.

docs/source/format/CDataInterfaceStatistics.rst

kou · 2024-08-07T02:39:12Z

docs/source/format/CDataInterfaceStatistics.rst

+The ``ARROW`` pattern is a reserved namespace for pre-defined
+statistics keys. User-defined statistics must not use it.
+
+Here are pre-defined statistics keys:


They are the same as ADBC's one: https://github.com/apache/arrow-adbc/blob/05fa60d643c66b572d426ab28aa78fc52e9520e8/c/include/arrow-adbc/adbc.h#L524-L570

kou · 2024-08-07T02:49:17Z

I was just slightly confused for one second that the second mapping is the value items in the first mapping.

Ah, it makes sense. I'll improve it. Thanks.

kou · 2024-08-07T03:57:24Z

@github-actions crossbow submit preview-docs

docs/source/format/CDataInterfaceStatistics.rst

…ough the Arrow C data interface

Co-authored-by: Ian Cook <ianmcook@gmail.com>

### Rationale for this change Statistics schema for Arrow C data interface (GH-43553) is complex because it uses nested types (struct, map and union). So reusable implementation to make statistics array is useful. ### What changes are included in this PR? `arrow::RecordBatch::MakeStatisticsArray()` is a convenient function that converts `arrow::ArrayStatistics` in a `arrow::RecordBatch` to `arrow::Array` for the Arrow C data interface. ### Are these changes tested? Yes. ### Are there any user-facing changes? Yes. * GitHub Issue: #44010 Authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

kou marked this pull request as draft August 5, 2024 09:28

github-actions bot added Component: Documentation awaiting committer review Awaiting committer review labels Aug 5, 2024

ianmcook reviewed Aug 5, 2024

View reviewed changes

docs/source/format/CDataInterfaceStatistics.rst Outdated Show resolved Hide resolved

github-actions bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Aug 5, 2024

ianmcook reviewed Aug 5, 2024

View reviewed changes

docs/source/format/CDataInterfaceStatistics.rst Outdated Show resolved Hide resolved

ianmcook reviewed Aug 5, 2024

View reviewed changes

ianmcook reviewed Aug 6, 2024

View reviewed changes

docs/source/format/CDataInterfaceStatistics.rst Outdated Show resolved Hide resolved

ianmcook reviewed Aug 6, 2024

View reviewed changes

docs/source/format/CDataInterfaceStatistics.rst Outdated Show resolved Hide resolved

ianmcook reviewed Aug 6, 2024

View reviewed changes

docs/source/format/CDataInterfaceStatistics.rst Outdated Show resolved Hide resolved

ianmcook reviewed Aug 6, 2024

View reviewed changes

docs/source/format/CDataInterfaceStatistics.rst Outdated Show resolved Hide resolved

ianmcook reviewed Aug 6, 2024

View reviewed changes

docs/source/format/CDataInterfaceStatistics.rst Outdated Show resolved Hide resolved

ianmcook reviewed Aug 6, 2024

View reviewed changes

docs/source/format/CDataInterfaceStatistics.rst Outdated Show resolved Hide resolved

ianmcook reviewed Aug 6, 2024

View reviewed changes

docs/source/format/CDataInterfaceStatistics.rst Outdated Show resolved Hide resolved

kou commented Aug 7, 2024

View reviewed changes

github-actions bot added awaiting change review Awaiting change review awaiting changes Awaiting changes and removed awaiting changes Awaiting changes awaiting change review Awaiting change review labels Aug 7, 2024

github-actions bot removed the awaiting changes Awaiting changes label Aug 7, 2024

github-actions bot added the awaiting changes Awaiting changes label Aug 22, 2024

kou mentioned this pull request Aug 26, 2024

GH-43631: [C][Format] Add ArrowAsyncDeviceStreamHandler interface #43632

Merged

kou force-pushed the docs-statistics-c-data-interface branch from 64dea6b to 3cdb559 Compare August 27, 2024 00:09

github-actions bot added awaiting change review Awaiting change review awaiting changes Awaiting changes and removed awaiting changes Awaiting changes awaiting change review Awaiting change review labels Aug 27, 2024

kou force-pushed the docs-statistics-c-data-interface branch from 3cdb559 to e22ec6c Compare September 9, 2024 02:25

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Sep 9, 2024

kou commented Sep 24, 2024

View reviewed changes

docs/source/format/CDataInterfaceStatistics.rst Show resolved Hide resolved

github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Sep 24, 2024

kou commented Sep 26, 2024

View reviewed changes

docs/source/format/CDataInterfaceStatistics.rst Outdated Show resolved Hide resolved

kou mentioned this pull request Sep 30, 2024

GH-44010: [C++] Add arrow::RecordBatch::MakeStatisticsArray() #44252

Merged

kou and others added 9 commits September 30, 2024 17:17

apacheGH-38837: [Format] Add the specification to pass statistics thr…

c173e60

…ough the Arrow C data interface

Improve wording

f11b459

Co-authored-by: Ian Cook <ianmcook@gmail.com>

Improve schema description

e0e4321

Link to ADBC's documentation

44db109

Add can we use this for the Arrow IPC format

0dbc8f6

Remove outer map

b5e017e

Add missing top-level struct

354bcb2

Add ARROW:null_count:{exact,approximate}

115a9b9

Use "statistics" not "values"

9087113

kou force-pushed the docs-statistics-c-data-interface branch from e22ec6c to 9087113 Compare September 30, 2024 08:20

github-actions bot added awaiting change review Awaiting change review awaiting changes Awaiting changes and removed awaiting changes Awaiting changes awaiting change review Awaiting change review labels Sep 30, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface #43553

GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface #43553

kou commented Aug 5, 2024 •

edited by github-actions bot

Loading

kou commented Aug 5, 2024

github-actions bot commented Aug 5, 2024

kou commented Aug 5, 2024

github-actions bot commented Aug 5, 2024

ianmcook Aug 5, 2024

kou Aug 7, 2024

ianmcook Aug 5, 2024

kou Aug 7, 2024

ianmcook commented Aug 5, 2024

pdet commented Aug 6, 2024

kou left a comment

kou Aug 7, 2024

kou Aug 7, 2024

kou Aug 7, 2024

kou commented Aug 7, 2024

kou commented Aug 7, 2024

		* Provide a common way to pass statistics that can be used for
		other interfaces such Arrow Flight too.

		For example, ADBC has the statistics related APIs. This specification
		doesn't replace them.

GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface #43553

Are you sure you want to change the base?

GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface #43553

Conversation

kou commented Aug 5, 2024 • edited by github-actions bot Loading

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

kou commented Aug 5, 2024

github-actions bot commented Aug 5, 2024

kou commented Aug 5, 2024

github-actions bot commented Aug 5, 2024

ianmcook Aug 5, 2024

Choose a reason for hiding this comment

kou Aug 7, 2024

Choose a reason for hiding this comment

ianmcook Aug 5, 2024

Choose a reason for hiding this comment

kou Aug 7, 2024

Choose a reason for hiding this comment

ianmcook commented Aug 5, 2024

pdet commented Aug 6, 2024

kou left a comment

Choose a reason for hiding this comment

kou Aug 7, 2024

Choose a reason for hiding this comment

kou Aug 7, 2024

Choose a reason for hiding this comment

kou Aug 7, 2024

Choose a reason for hiding this comment

kou commented Aug 7, 2024

kou commented Aug 7, 2024

kou commented Aug 5, 2024 •

edited by github-actions bot

Loading