GH-47838: [C++][Parquet] Set Variant specification version to 1 to align with the variant spec #47835

aihuaxu · 2025-10-16T17:52:57Z

Rationale for this change

According to the Variant specification, the specification_version field must be set to 1 to indicate Variant encoding version 1. Currently, this field defaults to 0, which violates the specification. Parquet readers that strictly enforce specification version validation will fail to read files containing Variant types.

What changes are included in this PR?

The change includes defaulting the specification version to 1.

Are these changes tested?

The change is covered by unit test.

Are there any user-facing changes?

The Parquet files produced the variant logical type annotation VARIANT(1).

Schema:
message schema {
  optional group V (VARIANT(1)) = 1 {
    required binary metadata;
    required binary value;
  }
}

GitHub Issue: [C++][Parquet] Variant specification version should be set 1 to align with the variant spec rather than 0 #47838

github-actions · 2025-10-16T17:53:22Z

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

aihuaxu · 2025-10-16T17:53:41Z

@wgtmac Can you take a look?

cpp/src/parquet/schema_test.cc

github-actions · 2025-10-16T20:57:01Z

⚠️ GitHub issue #47838 has been automatically assigned in GitHub to PR creator.

cpp/src/parquet/parquet.thrift

wgtmac

+1

I'm fine with the current change for now. Eventually we may need to add setter and getter to VariantLogicalType for specification_version and validating its value.

cc @pitrou @raulcd

pitrou · 2025-10-17T08:19:42Z

Parquet readers that strictly enforce specification version validation will fail to read files containing Variant types.

Why is it not the case for our own reader?

aihuaxu · 2025-10-17T16:56:13Z

Parquet readers that strictly enforce specification version validation will fail to read files containing Variant types.

Why is it not the case for our own reader?

You mean Parquet C++ reader? Currently Parquet C++ reader hasn't implemented Variant read/write yet. The only thing it added is to support Variant logical type and we are not doing the check. Currently Variant spec version is version 1 (and this is the only version) and the other readers may/may not add the check.

aihuaxu · 2025-10-17T18:38:01Z

@wgtmac and @pitrou I added specification_version to VariantLogicalType. Please take another look. Thanks.

raulcd · 2025-10-17T18:40:21Z

You mean Parquet C++ reader? Currently Parquet C++ reader hasn't implemented Variant read/write yet. The only thing it added is to support Variant logical type and we are not doing the check. Currently Variant spec version is version 1 (and this is the only version) and the other readers may/may not add the check.

Sorry, because I might be missing something obvious, I am not too familiar with this part of the codebase, but if we haven't implemented Parquet C++ Variant write yet, I am not sure I understand how can a user would be able to create Variant files with a logical type annotation that will be incorrect with Parquet C++ if we release without this fix.

aihuaxu · 2025-10-17T22:17:05Z

You mean Parquet C++ reader? Currently Parquet C++ reader hasn't implemented Variant read/write yet. The only thing it added is to support Variant logical type and we are not doing the check. Currently Variant spec version is version 1 (and this is the only version) and the other readers may/may not add the check.

Sorry, because I might be missing something obvious, I am not too familiar with this part of the codebase, but if we haven't implemented Parquet C++ Variant write yet, I am not sure I understand how can a user would be able to create Variant files with a logical type annotation that will be incorrect with Parquet C++ if we release without this fix.

The engines will implement the reader/writer parts but will use the variant type defined in Arrow Parquet. That would cause the engines to write incorrect annotation. That's what I'm seeing internally.

raulcd · 2025-10-20T07:23:13Z

We haven't been able to publish Python 3.14 wheels for PyArrow and the community is eager to get those. @pitrou @wgtmac can you help with this issue?
Is the issue a blocker? If it is, can we get this fixed so we can generate a new RC soon, if it is not, can you share the thoughts on the ML thread so we can proceed with RC0?

cpp/src/parquet/types.cc

pitrou · 2025-10-20T07:33:52Z

cpp/src/parquet/types.cc

+}
+
+std::shared_ptr<const LogicalType> VariantLogicalType::Make(const int8_t specVersion) {
+  auto logical_type = std::shared_ptr<VariantLogicalType>(new VariantLogicalType());


Can we use std::make_shared? It's more terse and more efficient.

Has this been addressed or resolved?

Unfortunately the VariantLogicalType constructor is private, so std::make_shared cannot work with it.

cpp/src/parquet/types.h

pitrou · 2025-10-20T07:36:06Z

We haven't been able to publish Python 3.14 wheels for PyArrow and the community is eager to get those. @pitrou @wgtmac can you help with this issue?

You mean help with the 3.14 wheels? I'm not sure I understand your message correctly.

Is the issue a blocker?

I would not call it a blocker entirely, but we would certainly rather have it.

raulcd · 2025-10-20T07:38:20Z

You mean help with the 3.14 wheels? I'm not sure I understand your message correctly.

No, sorry, I meant this issue which is currently holding the release and holding things like publishing the Python 3.14 wheels.

pitrou

+1 now that comments are addressed. Let's wait for CI.

pitrou · 2025-10-20T11:55:17Z

CI failures are unrelated, I'll merge. Thanks for spotting and fixing this @aihuaxu !

pitrou · 2025-10-20T11:56:05Z

@raulcd I'll let you make the final decision, but it would be nice if a new RC could be issued with this fix.

raulcd · 2025-10-20T12:07:14Z

I'll create a new RC. Thanks everyone for the quick fix!

…ign with the variant spec (#47835) ### Rationale for this change According to the [Variant specification](https://github.com/apache/parquet-format/blob/master/VariantEncoding.md), the specification_version field must be set to 1 to indicate Variant encoding version 1. Currently, this field defaults to 0, which violates the specification. Parquet readers that strictly enforce specification version validation will fail to read files containing Variant types. <img width="624" height="185" alt="image" src="https://github.com/user-attachments/assets/b0f1deb9-0301-4b94-a472-17fd9cc0df5d" /> ### What changes are included in this PR? The change includes defaulting the specification version to 1. ### Are these changes tested? The change is covered by unit test. ### Are there any user-facing changes? The Parquet files produced the variant logical type annotation `VARIANT(1)`. ``` Schema: message schema { optional group V (VARIANT(1)) = 1 { required binary metadata; required binary value; } } ``` * GitHub Issue: #47838 Lead-authored-by: Aihua <aihua.xu@snowflake.com> Co-authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Antoine Pitrou <antoine@python.org>

wgtmac · 2025-10-20T13:18:12Z

Thanks @pitrou and @raulcd!

conbench-apache-arrow · 2025-10-20T17:53:22Z

After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit 5f616db.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 6 possible false positives for unstable benchmarks that are known to sometimes produce them.

… to align with the variant spec (apache#47835) ### Rationale for this change According to the [Variant specification](https://github.com/apache/parquet-format/blob/master/VariantEncoding.md), the specification_version field must be set to 1 to indicate Variant encoding version 1. Currently, this field defaults to 0, which violates the specification. Parquet readers that strictly enforce specification version validation will fail to read files containing Variant types. <img width="624" height="185" alt="image" src="https://github.com/user-attachments/assets/b0f1deb9-0301-4b94-a472-17fd9cc0df5d" /> ### What changes are included in this PR? The change includes defaulting the specification version to 1. ### Are these changes tested? The change is covered by unit test. ### Are there any user-facing changes? The Parquet files produced the variant logical type annotation `VARIANT(1)`. ``` Schema: message schema { optional group V (VARIANT(1)) = 1 { required binary metadata; required binary value; } } ``` * GitHub Issue: apache#47838 Lead-authored-by: Aihua <aihua.xu@snowflake.com> Co-authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Antoine Pitrou <antoine@python.org>

aihuaxu requested a review from wgtmac as a code owner October 16, 2025 17:52

github-actions bot added Component: Parquet Component: C++ awaiting review Awaiting review labels Oct 16, 2025

sfc-gh-ebrossard reviewed Oct 16, 2025

View reviewed changes

cpp/src/parquet/schema_test.cc Outdated Show resolved Hide resolved

cpp/src/parquet/schema_test.cc Outdated Show resolved Hide resolved

aihuaxu changed the title ~~Set Variant specification version to 1 to align with the variant spec~~ GH-47838: [C++] Set Variant specification version to 1 to align with the variant spec Oct 16, 2025

github-actions bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Oct 16, 2025

wgtmac reviewed Oct 17, 2025

View reviewed changes

cpp/src/parquet/parquet.thrift Outdated Show resolved Hide resolved

aihuaxu changed the title ~~GH-47838: [C++] Set Variant specification version to 1 to align with the variant spec~~ GH-47838: [C++][Parquet] Set Variant specification version to 1 to align with the variant spec Oct 17, 2025

aihuaxu requested a review from wgtmac October 17, 2025 04:45

aihuaxu force-pushed the aixu-update-spec-version branch from a7c4d0d to 3450f1c Compare October 17, 2025 05:00

wgtmac approved these changes Oct 17, 2025

View reviewed changes

raulcd mentioned this pull request Oct 17, 2025

[CI][Ruby][Flight] flight-sql test fails on PRs with invalid SQL #47842

Open

aihuaxu force-pushed the aixu-update-spec-version branch from 4b65ada to 38a919d Compare October 17, 2025 18:37

pitrou requested changes Oct 20, 2025

View reviewed changes

sfc-gh-aixu added 3 commits October 20, 2025 09:45

Set Variant specification version to 1 to align with the variant spec

1fa5883

Test update

d56b188

Set specVersion in toThrift

3d6bc57

sfc-gh-aixu and others added 2 commits October 20, 2025 09:45

Add spec version to VariantLogicalType

5c21d65

Address review comments

cc1a45b

pitrou force-pushed the aixu-update-spec-version branch from 38a919d to cc1a45b Compare October 20, 2025 07:52

pitrou approved these changes Oct 20, 2025

View reviewed changes

pitrou merged commit 5f616db into apache:main Oct 20, 2025
111 of 160 checks passed

pitrou removed the awaiting committer review Awaiting committer review label Oct 20, 2025

pitrou mentioned this pull request Oct 20, 2025

[C++][Parquet] Variant specification version should be set 1 to align with the variant spec rather than 0 #47838

Closed

github-actions bot added the awaiting committer review Awaiting committer review label Oct 20, 2025

alamb mentioned this pull request Oct 28, 2025

Add Variant, Geometry, Geography types to implementation status page apache/parquet-site#123

Merged

GH-47838: [C++][Parquet] Set Variant specification version to 1 to align with the variant spec #47835

GH-47838: [C++][Parquet] Set Variant specification version to 1 to align with the variant spec #47835

Uh oh!

Conversation

aihuaxu commented Oct 16, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions bot commented Oct 16, 2025

Uh oh!

aihuaxu commented Oct 16, 2025

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Oct 16, 2025

Uh oh!

Uh oh!

wgtmac left a comment

Choose a reason for hiding this comment

Uh oh!

pitrou commented Oct 17, 2025

Uh oh!

aihuaxu commented Oct 17, 2025

Uh oh!

aihuaxu commented Oct 17, 2025

Uh oh!

raulcd commented Oct 17, 2025

Uh oh!

aihuaxu commented Oct 17, 2025

Uh oh!

raulcd commented Oct 20, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pitrou Oct 20, 2025

Choose a reason for hiding this comment

Uh oh!

reneleonhardt Oct 20, 2025

Choose a reason for hiding this comment

Uh oh!

pitrou Oct 20, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pitrou commented Oct 20, 2025

Uh oh!

raulcd commented Oct 20, 2025

Uh oh!

pitrou left a comment

Choose a reason for hiding this comment

Uh oh!

pitrou commented Oct 20, 2025

Uh oh!

Uh oh!

pitrou commented Oct 20, 2025

Uh oh!

raulcd commented Oct 20, 2025

Uh oh!

wgtmac commented Oct 20, 2025

Uh oh!

conbench-apache-arrow bot commented Oct 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

aihuaxu commented Oct 16, 2025 •

edited by github-actions bot

Loading