GH-37381: [Python][CI][Packaging] Enable ORC for Pyarrow on Appveyor and Windows wheels

[raulcd]

Rationale for this change

ORC on Windows wheels and CI should be enabled.

What changes are included in this PR?

Enabling it on CI + our Windows wheels

Are these changes tested?

On CI

Are there any user-facing changes?

Yes, currently Windows wheels do not have ORC enabled. They should in the future.
It is not a breaking change.

• Closes: [Python][CI][Packaging] Enable ORC on Windows Appveyor CI and Windows wheels for pyarrow #37381

[github-actions]

⚠️ GitHub issue #37381 has been automatically assigned in GitHub to PR creator.

[raulcd]

@github-actions crossbow submit wheel-windows*

[github-actions]

Revision: 433bf19

Submitted crossbow builds: ursacomputing/crossbow @ actions-f8e49ac3ec

Task	Status
wheel-windows-cp310-amd64
wheel-windows-cp311-amd64
wheel-windows-cp38-amd64
wheel-windows-cp39-amd64

[raulcd]

@github-actions crossbow submit wheel-windows-cp310-amd64

[github-actions]

Revision: 9c53912

Submitted crossbow builds: ursacomputing/crossbow @ actions-16139d3e7e

Task	Status
wheel-windows-cp310-amd64

[raulcd]

@github-actions crossbow submit wheel-windows-cp310-amd64

[github-actions]

Revision: 3f0433a

Submitted crossbow builds: ursacomputing/crossbow @ actions-44972c4679

Task	Status
wheel-windows-cp310-amd64

[raulcd]

@github-actions crossbow submit wheel-windows-cp310-amd64

[github-actions]

Revision: 539d09d

Submitted crossbow builds: ursacomputing/crossbow @ actions-1a3388f2a5

Task	Status
wheel-windows-cp310-amd64

[raulcd]

I am not sure why the wheel job is failing with (https://github.com/ursacomputing/crossbow/actions/runs/5997471725/job/16263918617#step:7:3778):

 C:/arrow/cpp/src/arrow/adapters/orc/adapter.cc(492): error C2100: illegal indirection (compiling source file C:\arrow-build\src\arrow\CMakeFiles\arrow_shared.dir\Unity\unity_26_cxx.cxx) [C:\arrow-build\src\arrow\arrow_shared.vcxproj]

The line that seems to fail is StripeInformation stripe_info({0, 0, 0, 0}); from the following:

    if (!include_indices.empty()) {
      RETURN_NOT_OK(SelectIndices(&opts, include_indices));
    }
    StripeInformation stripe_info({0, 0, 0, 0});
    RETURN_NOT_OK(SelectStripeWithRowNumber(&opts, current_row_, &stripe_info));
    ARROW_ASSIGN_OR_RAISE(auto schema, ReadSchema(opts));
    std::unique_ptr<liborc::RowReader> row_reader;

https://github.com/apache/arrow/blob/main/cpp/src/arrow/adapters/orc/adapter.cc#L492

@wgtmac any idea?

[wgtmac]

I am not sure why the wheel job is failing with (https://github.com/ursacomputing/crossbow/actions/runs/5997471725/job/16263918617#step:7:3778):

 C:/arrow/cpp/src/arrow/adapters/orc/adapter.cc(492): error C2100: illegal indirection (compiling source file C:\arrow-build\src\arrow\CMakeFiles\arrow_shared.dir\Unity\unity_26_cxx.cxx) [C:\arrow-build\src\arrow\arrow_shared.vcxproj]

The line that seems to fail is StripeInformation stripe_info({0, 0, 0, 0}); from the following:

    if (!include_indices.empty()) {
      RETURN_NOT_OK(SelectIndices(&opts, include_indices));
    }
    StripeInformation stripe_info({0, 0, 0, 0});
    RETURN_NOT_OK(SelectStripeWithRowNumber(&opts, current_row_, &stripe_info));
    ARROW_ASSIGN_OR_RAISE(auto schema, ReadSchema(opts));
    std::unique_ptr<liborc::RowReader> row_reader;

https://github.com/apache/arrow/blob/main/cpp/src/arrow/adapters/orc/adapter.cc#L492

@wgtmac any idea?

This may be related.
https://stackoverflow.com/questions/49807527/c-c2100-illegal-indirection-on-initialize-struct-variable

Could you please try StripeInformation stripe_info{0, 0, 0, 0}; ?

[raulcd]

@github-actions crossbow submit wheel-windows-cp310-amd64

[github-actions]

Revision: 626dea5

Submitted crossbow builds: ursacomputing/crossbow @ actions-2464a7f3cf

Task	Status
wheel-windows-cp310-amd64

[raulcd]

@github-actions crossbow submit wheel-windows-cp310-amd64

[github-actions]

Revision: 5ff8914

Submitted crossbow builds: ursacomputing/crossbow @ actions-d365f96774

Task	Status
wheel-windows-cp310-amd64

[raulcd]

@github-actions crossbow submit wheel-windows-*

[github-actions]

Revision: ce4fe1c

Submitted crossbow builds: ursacomputing/crossbow @ actions-0bc3675cbd

Task	Status
wheel-windows-cp310-amd64
wheel-windows-cp311-amd64
wheel-windows-cp38-amd64
wheel-windows-cp39-amd64

[jorisvandenbossche]

Looks good! (assuming the triggered builds will pass)

[wgtmac]

@jorisvandenbossche Do you have any idea on the backtrace: #36026 (comment) ?

[raulcd]

@jorisvandenbossche Do you have any idea on the backtrace: #36026 (comment) ?

I am not sure why but the Appveyor tests are successful and I've double checked the orc tests are being executed: https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow/builds/47906815#L3544

[jorisvandenbossche]

Unfortunately those python stacktraces are mostly useless (it only indicates the the segfault happens when actually scanning the data (with dataset.to_table()), and so eg not in the dataset discovery, but that would also have been my guess).
Ideally we would get an actual backtrace like I could get locally with gdb on linux (or enable catchsegv on CI), but I am not very familiar how to do that on Windows (and in CI).

[felipecrv]

Are the versions of protobuf used to build ORC and Arrow the same on Windows?

protocolbuffers/protobuf#435 (comment)

[raulcd]

@github-actions crossbow submit wheel-windows-*

[github-actions]

Revision: 6ccd89c

Submitted crossbow builds: ursacomputing/crossbow @ actions-d2cce94439

Task	Status
wheel-windows-cp310-amd64
wheel-windows-cp311-amd64
wheel-windows-cp312-amd64
wheel-windows-cp38-amd64
wheel-windows-cp39-amd64

[pitrou]

@raulcd Do you want to revive this?

[raulcd]

Closing this in favor of: #40609