fix: Update the pyarrow to latest v14.0.1 regarding the CVE-2023-47248.

[shuchu]

What this PR does / why we need it:
Update the pyarrow to latest version v14.0.1 which has the fix for CVE-2023-47248

Fixes #3832

[shuchu]

A little bit worried about the unit test coverage.

please be aware that I unpin the pyarrow version.

py3.8-requirements.txt and py3.8-ci-requirements.txt were updated manually. (regarding the DASK version issue for python 3.8)

[achals]

I think this is incorrect

[shuchu]

you are right, I need to create a python 3.10 venv and run the command from Makefile. let me fix this.

[shuchu]

fixed, let's see the testing results.

[shuchu]

seems integration test failed...

[shuchu]

21 fails, the most frequent error is about the wrong format of Timestamp: google.api_core.exceptions.BadRequest: 400 Error while reading data, error message: Invalid timestamp microseconds value 1700011424237000000 of logical type NONE; in column 'created'

let me dig into it and see what's the root cause

[shuchu]

1, The Google's BigQuery api only accepts "ms" resolution for timestamp, while the Pyarrow.parquet.write_table() will maintain the resolution to the exact original resolution which is "ns" by default.
https://arrow.apache.org/docs/python/generated/pyarrow.parquet.write_table.html

[shuchu]

I meet a very interesting problem. I only update the Pyarrow version and snowflake api, the integration test results show me that the timestamp range is error while running Redshift SQL query.

{ error:  Timestamp out of range.\n  
              code:      8001\n  }

it happens while run "get_historical_features()" and the timestamp range were inferenced from the "entity_df":
as in redshift.py::_get_entity_df_event_timestamp_range().
f"SELECT MIN({entity_df_event_timestamp_col}) AS min, MAX({entity_df_event_timestamp_col}) AS max "

[shuchu]

please do not merge this PR. @sudohainguyen
It's in a mess status and is for debugging only now.

[sudohainguyen]

No worries, looking forward to seeing this works

[shuchu]

Finally, I found the fix way. It's about the setting of parameter "coerce_timestamps" of "pyarrow.parquet.write_table".

Let me close this PR and create a clean new one.

[sudohainguyen]

Great @shuchu !!