Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface #43553

Closed
wants to merge 19 commits into from
Closed

GH-38837: [Format] Add the specification to pass statistics through the Arrow C data interface #43553

wants to merge 19 commits into from
+343 −0

Conversation

kou
Copy link
Member

@kou kou commented Aug 5, 2024 •
edited by github-actions bot
Loading

Rationale for this change

Statistics are useful for fast query processing. Many query engines
use statistics to optimize their query plan.

Apache Arrow format doesn't have statistics but other formats that can
be read as Apache Arrow data may have statistics. For example, Apache
Parquet C++ can read Apache Parquet file as Apache Arrow data and
Apache Parquet file may have statistics.

One of the Arrow C data interface use cases is the following:

  1. Module A reads Apache Parquet file as Apache Arrow data
  2. Module A passes the read Apache Arrow data to module B through the
    Arrow C data interface
  3. Module B processes the passed Apache Arrow data

If module A can pass the statistics associated with the Apache Parquet
file to module B through the Arrow C data interface, module B can use
the statistics to optimize its query plan.

What changes are included in this PR?

Add the specification to pass statistics through the Arrow C data interface based on the discussion on the dev@ mailing list: https://lists.apache.org/thread/z0jz2bnv61j7c6lbk7lympdrs49f69cx

Are these changes tested?

Yes.

Are there any user-facing changes?

Yes.

@kou kou marked this pull request as draft August 5, 2024 09:28
@kou
Copy link
Member Author

kou commented Aug 5, 2024

@github-actions crossbow submit preview-docs

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 5, 2024

⚠️ GitHub issue #38837 has been automatically assigned in GitHub to PR creator.

@kou
Copy link
Member Author

kou commented Aug 5, 2024

I'm not a native English speaker. Wording suggestions are very welcome.

I'll add examples after I implement a convenient API to C++.

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 5, 2024

Revision: 22336f4

Submitted crossbow builds: ursacomputing/crossbow @ actions-28c2a45b3d

Task Status
preview-docs GitHub Actions

@github-actions github-actions bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Aug 5, 2024
ianmcook
ianmcook reviewed Aug 5, 2024
View reviewed changes
docs/source/format/CDataInterfaceStatistics.rst
Comment on lines +56 to +76
* Provide a common way to pass statistics that can be used for
other interfaces such Arrow Flight too.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about the Arrow IPC format? Can you add a sentence here that explains why we do not recommend using this to pass statistics over Arrow IPC?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. This may fit the Arrow IPC format. (A producer sends data and statistics as 2 separated the Arrow IPC format data.)

But the Arrow IPC format can use more approaches. For example, the Arrow IPC format can have metadata for each record batch data: https://arrow.apache.org/docs/format/Columnar.html#encapsulated-message-format (The Arrow C data can't have metadata for ArrowArray.)
The Arrow IPC format can be used with other mechanisms such as Arrow Flight and ADBC.

So this may not be the best approach for the Arrow IPC format. We should discuss this use case with the Arrow IPC format separately.

I'll add something to here.

ianmcook
ianmcook reviewed Aug 5, 2024
View reviewed changes
docs/source/format/CDataInterfaceStatistics.rst Outdated
Comment on lines 59 to 60
For example, ADBC has the statistics related APIs. This specification
doesn't replace them.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe link to https://arrow.apache.org/adbc/current/format/specification.html#statistics

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I should have done it.

@ianmcook
Copy link
Member

ianmcook commented Aug 5, 2024

@pdet please take a look and add comments if you have any, thanks!

@pdet
Copy link

pdet commented Aug 6, 2024

The format and contents LGTM! I was just slightly confused for one second that the second mapping is the value items in the first mapping.

@Tmonster I think the proposed Statistics keys already cover the table statistics we use/produce. Could you double-check?

kou
kou commented Aug 7, 2024
View reviewed changes
Copy link
Member Author

@kou kou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ianmcook Thanks for your suggestions! I've merged all of them!

docs/source/format/CDataInterfaceStatistics.rst
Comment on lines +56 to +76
* Provide a common way to pass statistics that can be used for
other interfaces such Arrow Flight too.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. This may fit the Arrow IPC format. (A producer sends data and statistics as 2 separated the Arrow IPC format data.)

But the Arrow IPC format can use more approaches. For example, the Arrow IPC format can have metadata for each record batch data: https://arrow.apache.org/docs/format/Columnar.html#encapsulated-message-format (The Arrow C data can't have metadata for ArrowArray.)
The Arrow IPC format can be used with other mechanisms such as Arrow Flight and ADBC.

So this may not be the best approach for the Arrow IPC format. We should discuss this use case with the Arrow IPC format separately.

I'll add something to here.

docs/source/format/CDataInterfaceStatistics.rst Outdated
Comment on lines 59 to 60
For example, ADBC has the statistics related APIs. This specification
doesn't replace them.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I should have done it.

docs/source/format/CDataInterfaceStatistics.rst Outdated Show resolved Hide resolved
docs/source/format/CDataInterfaceStatistics.rst
The ``ARROW`` pattern is a reserved namespace for pre-defined
statistics keys. User-defined statistics must not use it.

Here are pre-defined statistics keys:
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They are the same as ADBC's one: https://github.com/apache/arrow-adbc/blob/05fa60d643c66b572d426ab28aa78fc52e9520e8/c/include/arrow-adbc/adbc.h#L524-L570

@github-actions github-actions bot added awaiting change review Awaiting change review awaiting changes Awaiting changes and removed awaiting changes Awaiting changes awaiting change review Awaiting change review labels Aug 7, 2024
@kou
Copy link
Member Author

kou commented Aug 7, 2024

I was just slightly confused for one second that the second mapping is the value items in the first mapping.

Ah, it makes sense. I'll improve it. Thanks.

@kou
Copy link
Member Author

kou commented Aug 7, 2024

@github-actions crossbow submit preview-docs

@github-actions github-actions bot removed the awaiting changes Awaiting changes label Aug 7, 2024
kou and others added 7 commits December 11, 2024 15:20
@kou
Add a C++ example
9f2f017
@kou
Add experimental warning
4644418
@kou
index -> type code
595d916
@kou
Clarify wording
5fbddf4
@kou
Add user-defined statistics example
6815b99
@kou
Fix a typo
f744c80
@kou
Update
ed0cbe2 
* Add the original DuckDB use case
* Add TODOs
* Clarify "column index": It uses the flattened index
* Clarify statistics target
* Use concrete data not C++ for examples
@kou kou force-pushed the docs-statistics-c-data-interface branch from 6cf71e0 to ed0cbe2 Compare December 11, 2024 07:58
@github-actions github-actions bot added awaiting changes Awaiting changes awaiting change review Awaiting change review and removed awaiting change review Awaiting change review awaiting changes Awaiting changes labels Dec 11, 2024
@github-actions github-actions bot added awaiting change review Awaiting change review awaiting changes Awaiting changes and removed awaiting changes Awaiting changes awaiting change review Awaiting change review labels Dec 11, 2024
@kou
Copy link
Member Author

kou commented Dec 11, 2024

I think the C Data Interface framing came from the original use case (which I would like to not lose sight of); allowing engines like DuckDB to have a way to get statistics when given a C Data Interface stream, so that they can properly do query planning. Even if we reframe this as just a definition for the format of the statistics it might be good to mention the original use case so there's proper context, to help commenters who aren't familiar with query planning.

I've added the original DuckDB use case.

DuckDB may be able to get statistics without ArrowArrayStream (DuckDB may be able to call separate API to get statistics) because duckdb::TableFunction has table_function_cardinality_t cardinality and table_statistics_t statistics.
See also: https://github.com/duckdb/duckdb/blob/v1.1.3/src/function/table/arrow.cpp#L525-L527

I'll start a discussion on the mailing list tomorrow.
(I wanted to do it today...)

@github-actions github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Dec 11, 2024
@kou
Copy link
Member Author

kou commented Dec 12, 2024

The discussion thread: https://lists.apache.org/thread/b6chzlyn95rztoybs39b6olz907g12gj

kou added a commit to kou/arrow that referenced this pull request Dec 18, 2024
@kou
apacheGH-43553: [Format] Add the specification for statistics schema
5308f9f
@github-actions github-actions bot mentioned this pull request Dec 18, 2024
Open
@kou
Copy link
Member Author

kou commented Dec 18, 2024

Based on the mailing list discussion, we don't limit this specification to only the Arrow C data interface.
We standardize only statistics schema.

I close this in favor of GH-45058.

@kou kou closed this Dec 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@paleolimbot paleolimbot paleolimbot left review comments

@wgtmac wgtmac wgtmac left review comments

@pitrou pitrou pitrou left review comments

@lidavidm lidavidm lidavidm left review comments

@ianmcook ianmcook ianmcook left review comments

@felipecrv felipecrv felipecrv left review comments

Assignees

@kou kou

Labels
awaiting change review Awaiting change review Component: C++ Component: Documentation
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
10 participants
@kou @ianmcook @pdet @Tmonster @CurtHagenlocher @lidavidm @felipecrv @pitrou @wgtmac @paleolimbot