GH-37381: [Python][CI][Packaging] Enable ORC in Windows wheels and Appveyor CI

[wgtmac]

Rationale for this change

The pyarrow orc reader always crashes when it tries to create an internal orc reader. This is caused by failing to read tz database on the local host. This also disables windows wheel build when ARROW_ORC is turned on.

What changes are included in this PR?

Download IANA timezone database on the test host and explicitly setting TZDIR to make the CIs happy.

Are these changes tested?

Make sure all python wheel windows CIs pass.

Are there any user-facing changes?

No.

• GitHub Issue: [Python] BUG: Reading ORC segfaults on windows (if TZDIR isn't set) #36026
• GitHub Issue: [Python][CI] Support ORC in Windows wheels #33499
• GitHub Issue: [Python][CI][Packaging] Enable ORC on Windows Appveyor CI and Windows wheels for pyarrow #37381

[wgtmac]

C:\>curl https://cygwin.osuosl.org/noarch/release/tzdata/tzdata-2024a-1.tar.xz --output tzdata.tar.gz   || exit /B 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
 51  451k   51  234k    0     0   234k      0  0:00:01 --:--:--  0:00:01  300k
100  451k  100  451k    0     0   451k      0  0:00:01 --:--:--  0:00:01  466k
C:\>mkdir C:\Users\ContainerAdministrator\Downloads\test\tzdata 
C:\>tar --extract --file tzdata.tar.gz --directory C:\Users\ContainerAdministrator\Downloads\test\tzdata 
tar: Error opening archive: Can't initialize filter; unable to run program "xz -d -qq"

Do you know any approach to unzip tar.xz file on Windows in the CI build? @kou

[kou]

+1

[wgtmac]

@github-actions crossbow submit wheel-windows*

[github-actions]

Revision: 1a2634b

Submitted crossbow builds: ursacomputing/crossbow @ actions-2e59d7101d

Task	Status
wheel-windows-cp310-amd64
wheel-windows-cp311-amd64
wheel-windows-cp312-amd64
wheel-windows-cp38-amd64
wheel-windows-cp39-amd64

[wgtmac]

Revision: 1a2634b

Submitted crossbow builds: ursacomputing/crossbow @ actions-2e59d7101d

Task Status
wheel-windows-cp310-amd64
wheel-windows-cp311-amd64
wheel-windows-cp312-amd64
wheel-windows-cp38-amd64
wheel-windows-cp39-amd64

Step 6/8 : RUN choco install -r -y --no-progress python --version=%PYTHON_VERSION%
 ---> Running in 99f097760eef
Installing the following packages:
python
By installing you accept licenses for the packages.
python not installed. An error occurred during installation:
 The remote server returned an error: (504) Gateway Timeout. Gateway Time-out
python package files install completed. Performing other installation steps.
The install of python was NOT successful.
python not installed. An error occurred during installation:
 The remote server returned an error: (504) Gateway Timeout. Gateway Time-out

Chocolatey installed 0/1 packages. 1 packages failed.
 See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).

Failures
 - python (exited 1) - python not installed. An error occurred during installation:
 The remote server returned an error: (504) Gateway Timeout. Gateway Time-out
The command 'cmd /S /C choco install -r -y --no-progress python --version=%PYTHON_VERSION%' returned a non-zero code: 1
Error: docker build --build-arg BUILDKIT_INLINE_CACHE=1 --build-arg python=3.9 -f D:\a\crossbow\crossbow\arrow\ci/docker/python-wheel-windows-test-vs2019.dockerfile -t ghcr.io/ursacomputing/arrow:python-3.9-wheel-windows-test-vs2019-2024-03-19 D:\a\crossbow\crossbow\arrow exited with non-zero exit code 1
docker pull ghcr.io/ursacomputing/arrow:python-3.9-wheel-windows-test-vs2019-2024-03-19 exited with non-zero exit code 1
Error: Process completed with exit code 1.

Failed with above logs. I don't think we need to rerun them.

[wgtmac]

Failure in AMD64 Ubuntu 20.04 R 4.3 Force-Tests true is also unrelated.

══ Failed tests ════════════════════════════════════════════════════════════════
── Error ('test-duckdb.R:286:5'): to_duckdb passing a connection ───────────────
Error in `unclass(unclass(table_four)$lazy_query$x)$table`: $ operator is invalid for atomic vectors

[ FAIL 1 | WARN 0 | SKIP 22 | PASS 8265 ]
Error: Test failures
Execution halted

1 error ✖ | 0 warnings ✔ | 2 notes ✖
1
Error: `docker-compose --file /home/runner/work/arrow/arrow/docker-compose.yml run --rm -e TZ=MART -e ARROW_R_FORCE_TESTS=true ubuntu-r` exited with a non-zero exit code 1, see the process log above.

[wgtmac]

@kou Do you want to take another pass?

[kou]

Thanks!

[jorisvandenbossche]

I am not sure that we should enable ORC in the wheels if this essentially always segfaults (on first use, or before you know what to do) for users.

At least we could add a check on our side, if on windows, that the path is available, and if not raise an informative error to the user instead of segfaulting

[pitrou]

Right. It seems that the ORC project should first make it possible to use without the timezone data files.

[wgtmac]

This PR aims to enable ORC in the wheels CI in case there is any new regression. I will work on the ORC side to make it robust as much as possible.

[jorisvandenbossche]

For CI we also have Appveyor, which was enabled here as well. But the wheels is what we deliver to our users. If we enable ORC there, I think we need to make this more robust on our side on the short term (or disable it again in the wheels until this is fixed in ORC and we use that version)

[wgtmac]

But the wheels is what we deliver to our users.

Sorry I'm not familiar with that. It sounds that we need to revert this change at least for wheels.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 0eefb07.

There were 8 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2024-03-20 14:37:21Z
  • file-read (R) with compression=uncompressed, dataset=fanniemae_2016Q4, file_type=feather, language=R, output_type=dataframe
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-20, scale_factor=1
• and 6 more (see the report linked below)

The full Conbench report has more details. It also includes information about 3 possible false positives for unstable benchmarks that are known to sometimes produce them.