Use ArrowScan.to_table to replace project_table #1180

JE-Chen · 2024-09-17T06:01:32Z

Use ArrowScan.to_table to replace project_table in these files:
- pyiceberg\table_init_.py
- pyiceberg\io\pyarrow.py
- pyiceberg\test_pyarrow.py

* Use ArrowScan.to_table to replace project_table on these file: ** pyiceberg\table\__init__.py ** pyiceberg\io\pyarrow.py ** pyiceberg\test_pyarrow.py

Replace all remaining of project_table using ArrowScan.to_table

sungwy · 2024-09-17T14:38:29Z

Hi @JE-Chen thank you for putting together this PR! It looks like these were missed in the initial refactoring. Running the CI now.

Fix format

Modify by ruff

JE-Chen · 2024-09-17T16:08:15Z

Hi @sungwy I have already fixed the problem that ruff found.

kevinjqliu

Thanks for the contribution!

Looks like this captures all the project_table usage https://github.com/search?q=repo%3Aapache%2Ficeberg-python%20project_table&type=code

Let's double check the log for make test and make test-integration to make sure there's no deprecation warnings related to project_table

JE-Chen · 2024-09-18T11:04:57Z

There are make test-integration logs that the CI test generated.
No warnings found on pyiceberg/table/init.py & tests/io/test_pyarrow.py.

=============================== warnings summary ===============================
tests/integration/test_add_files.py: 42 warnings
tests/integration/test_deletes.py: 27 warnings
tests/integration/test_inspect_table.py: 24 warnings
tests/integration/test_partition_evolution.py: 140 warnings
tests/integration/test_reads.py: 54 warnings
tests/integration/test_rest_schema.py: 91 warnings
tests/integration/test_snapshot_operations.py: 4 warnings
tests/integration/test_writes/test_partitioned_writes.py: 175 warnings
tests/integration/test_writes/test_writes.py: 128 warnings
== 809 passed, 7 skipped, 2840 deselected, 1441 warnings in 295.87s (0:04:55) ==

make test No warnings found on pyiceberg/table/init.py & tests/io/test_pyarrow.py too.

=============================== warnings summary ===============================
tests/integration/test_add_files.py: 42 warnings
tests/integration/test_deletes.py: 27 warnings
tests/integration/test_inspect_table.py: 24 warnings
tests/integration/test_partition_evolution.py: 140 warnings
tests/integration/test_reads.py: 54 warnings
tests/integration/test_rest_schema.py: 91 warnings
tests/integration/test_snapshot_operations.py: 4 warnings
tests/integration/test_writes/test_partitioned_writes.py: 175 warnings
tests/integration/test_writes/test_writes.py: 128 warnings
== 809 passed, 7 skipped, 2840 deselected, 1441 warnings in 283.39s (0:04:43) ==

Log of make test-integration on my VM has so many errors. Any ideas? (It succeeds on CI Test).

$ make test-integration
docker compose -f dev/docker-compose-integration.yml kill
WARN[0000] /home/jeffrey/Desktop/GItHub_Project/iceberg-python/dev/docker-compose-integration.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion 
[+] Killing 5/5
 ✔ Container pyiceberg-minio  Killed                                                                                                                                                         0.3s 
 ✔ Container pyiceberg-mc     Killed                                                                                                                                                         0.1s 
 ✔ Container pyiceberg-rest   Killed                                                                                                                                                         0.2s 
 ✔ Container hive             Killed                                                                                                                                                         0.2s 
 ✔ Container pyiceberg-spark  Killed                                                                                                                                                         0.3s 
docker compose -f dev/docker-compose-integration.yml rm -f
WARN[0000] /home/jeffrey/Desktop/GItHub_Project/iceberg-python/dev/docker-compose-integration.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion 
Going to remove pyiceberg-spark, pyiceberg-mc, pyiceberg-rest, hive, pyiceberg-minio
[+] Removing 5/0
 ✔ Container pyiceberg-minio  Removed                                                                                                                                                        0.0s 
 ✔ Container pyiceberg-spark  Removed                                                                                                                                                        0.0s 
 ✔ Container pyiceberg-rest   Removed                                                                                                                                                        0.0s 
 ✔ Container pyiceberg-mc     Removed                                                                                                                                                        0.0s 
 ✔ Container hive             Removed                                                                                                                                                        0.0s 
docker compose -f dev/docker-compose-integration.yml up -d
WARN[0000] /home/jeffrey/Desktop/GItHub_Project/iceberg-python/dev/docker-compose-integration.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion 
[+] Running 5/5
 ✔ Container pyiceberg-minio  Started                                                                                                                                                        0.5s 
 ✔ Container hive             Started                                                                                                                                                        0.4s 
 ✔ Container pyiceberg-rest   Started                                                                                                                                                        0.7s 
 ✔ Container pyiceberg-mc     Started                                                                                                                                                        0.9s 
 ✔ Container pyiceberg-spark  Started                                                                                                                                                        1.0s 
sleep 10
docker compose -f dev/docker-compose-integration.yml cp ./dev/provision.py spark-iceberg:/opt/spark/provision.py
WARN[0000] /home/jeffrey/Desktop/GItHub_Project/iceberg-python/dev/docker-compose-integration.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion 
[+] Copying 1/0
 ✔ pyiceberg-spark copy ./dev/provision.py to pyiceberg-spark:/opt/spark/provision.py Copied                                                                                                 0.0s 
docker compose -f dev/docker-compose-integration.yml exec -T spark-iceberg ipython ./provision.py
WARN[0000] /home/jeffrey/Desktop/GItHub_Project/iceberg-python/dev/docker-compose-integration.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion 
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
24/09/18 11:02:25 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Could not initialize FileIO: pyiceberg.io.pyarrow.PyArrowFileIO
Could not initialize FileIO: pyiceberg.io.pyarrow.PyArrowFileIO                 
Could not initialize FileIO: pyiceberg.io.pyarrow.PyArrowFileIO
Could not initialize FileIO: pyiceberg.io.pyarrow.PyArrowFileIO
poetry run pytest tests/ -v -m integration 

====================================================================================== test session starts =======================================================================================
platform linux -- Python 3.11.9, pytest-7.4.4, pluggy-1.5.0 -- /home/jeffrey/Desktop/GItHub_Project/iceberg-python/venv/bin/python
cachedir: .pytest_cache
rootdir: /home/jeffrey/Desktop/GItHub_Project/iceberg-python
configfile: pyproject.toml
plugins: lazy-fixture-0.6.3, mock-3.14.0, requests-mock-1.12.1, checkdocs-2.10.1
collected 3640 items / 2840 deselected / 800 selected 

... tests ...
====================================================== 501 passed, 7 skipped, 2840 deselected, 731 warnings, 292 errors in 99.24s (0:01:39) ======================================================

sungwy · 2024-09-18T13:09:45Z

Log of make test-integration on my VM has so many errors. Any ideas? (It succeeds on CI Test).

What kind of errors are you running into @JE-Chen ? Could you provide us an example?

JE-Chen · 2024-09-18T13:46:33Z

steps

Open Pycharm
Open iceberg project
Create new venv using python 3.11.9
pip install poetry
make install
make test-integration
Get this message four time Could not initialize FileIO: pyiceberg.io.pyarrow.PyArrowFileIO
Start run tests (all docker service is running) and get so many error (= 38 failed, 755 passed, 7 skipped, 2840 deselected, 1431 warnings in 409.59s (0:06:49) =)

platform linux -- Python 3.11.9, pytest-7.4.4, pluggy-1.5.0 -- /home/jeffrey/Desktop/GItHub_Project/iceberg-python/venv/bin/python
cachedir: .pytest_cache
rootdir: /home/jeffrey/Deskt501  38 failed, 755 passed, 7 skipped, 2840 deselected, 1431 warnings in 409.59s (0:06:49)

platform linux -- Python 3.11.9, pytest-7.4.4, pluggy-1.5.0 -- /home/jeffrey/Desktop/GItHub_Project/iceberg-python/venv/bin/python
cachedir: .pytest_cacheop/GItHub_Project/iceberg-python
configfile: pyproject.toml
plugins: lazy-fixture-0.6.3, mock-3.14.0, requests-mock-1.12.1, checkdocs-2.10.1

sungwy · 2024-09-18T13:53:54Z

Thank you for sending over the reproducible steps @JE-Chen - I do not use PyCharm for my development (I'm a VSCode user) so it might take a while for me to reproduce your setup.

Could not initialize FileIO: pyiceberg.io.pyarrow.PyArrowFileIO

iceberg-python/pyiceberg/io/__init__.py

Lines 320 to 322 in de47590

    
           except ModuleNotFoundError: 
        
               logger.warning("Could not initialize FileIO: %s", io_impl) 
        
               return None

This is a warning message and I'm familiar with seeing that in my integration tests as well, but it didn't result in any of the tests failing in my case. Are you able to share the verbose error trace for a failing test case, so we can investigate what its failing on?

JE-Chen · 2024-09-18T14:31:04Z

There are full error log.
https://gist.github.com/JE-Chen/d29aa4ab86d256cfe79788157ff83983

kevinjqliu · 2024-09-18T16:28:34Z

	Suppressed: org.apache.spark.util.TaskCompletionListenerException: Memory was leaked by query. Memory leaked: (3227648)
Allocator(toArrowBatchIterator) 0/3227648/3227648/9223372036854775807 (res/actual/peak/limit)

What VM are you running the integration tests on?

I have a theory that a previous change is causing memory leaks (see #1167). Can you try to revert it and run the integration test?
1971fcf

git revert 1971fcfe0875eeb200dbcb66f385e504cfad6609

JE-Chen · 2024-09-18T16:51:20Z

	Suppressed: org.apache.spark.util.TaskCompletionListenerException: Memory was leaked by query. Memory leaked: (3227648)
Allocator(toArrowBatchIterator) 0/3227648/3227648/9223372036854775807 (res/actual/peak/limit)
What VM are you running the integration tests on?

I have a theory that a previous change is causing memory leaks (see #1167). Can you try to revert it and run the integration test? 1971fcf
git revert 1971fcfe0875eeb200dbcb66f385e504cfad6609

The memory leak still occurred after reverting

VMware Workstation 17 Pro (Ubuntu 24.04.1) with these settings:

kevinjqliu · 2024-09-18T17:18:48Z

Gotcha, thanks! I think the CI runs in ubuntu, so it should mirror your own setup

JE-Chen · 2024-09-20T06:28:38Z

I’m trying to remove all docker images and re-run make-integration, and I noticed this.
So, I think the test failed because:
Can’t pull the python-integration image (access denied).

kevinjqliu · 2024-09-20T06:45:54Z

seems like its an issue with your docker installation.

iceberg-python/dev/docker-compose-integration.yml

Line 21 in 41a3c8e

image: python-integration

here's what I found for "VMware Workstation 17 Pro "
https://www.reddit.com/r/vmware/comments/167rurt/easy_way_to_run_docker_container_in_vmware/

JE-Chen · 2024-09-20T06:54:31Z

My Windows environment has the same pull issue.

(Docker plugin on Pycharm IDE)

(Run on powershell)

kevinjqliu · 2024-09-20T16:49:48Z

@JE-Chen I was able to run integration tests for this PR locally. And given that CI also pass, let's merge this and debug your VM environment issue separately.
How does that sound?

JE-Chen · 2024-09-20T16:51:48Z

@JE-Chen I was able to run integration tests for this PR locally. And given that CI also pass, let's merge this and debug your VM environment issue separately. How does that sound?

Sounds great.

kevinjqliu

LGTM, confirmed no more warnings

kevinjqliu · 2024-09-20T17:37:09Z

Thank you @JE-Chen for the contribution and @sungwy for the review

* Use ArrowScan.to_table to replace project_table * Use ArrowScan.to_table to replace project_table on these file: ** pyiceberg\table\__init__.py ** pyiceberg\io\pyarrow.py ** pyiceberg\test_pyarrow.py * Replace all remaining of project_table using ArrowScan.to_table Replace all remaining of project_table using ArrowScan.to_table * Fix format Fix format * Modify by ruff Modify by ruff

JE-Chen added 2 commits September 17, 2024 13:59

Use ArrowScan.to_table to replace project_table

b64c5d9

* Use ArrowScan.to_table to replace project_table on these file: ** pyiceberg\table\__init__.py ** pyiceberg\io\pyarrow.py ** pyiceberg\test_pyarrow.py

Replace all remaining of project_table using ArrowScan.to_table

10c7b5e

Replace all remaining of project_table using ArrowScan.to_table

JE-Chen added 2 commits September 17, 2024 23:49

Fix format

d5fa5f9

Fix format

Modify by ruff

4a47b61

Modify by ruff

kevinjqliu reviewed Sep 17, 2024

View reviewed changes

sungwy approved these changes Sep 18, 2024

View reviewed changes

kevinjqliu approved these changes Sep 20, 2024

View reviewed changes

kevinjqliu merged commit b8b2f66 into apache:main Sep 20, 2024
8 checks passed

kevinjqliu mentioned this pull request Sep 20, 2024

project_table is deprecated, remove references #1119

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use ArrowScan.to_table to replace project_table #1180

Use ArrowScan.to_table to replace project_table #1180

JE-Chen commented Sep 17, 2024 •

edited

Loading

sungwy commented Sep 17, 2024

JE-Chen commented Sep 17, 2024

kevinjqliu left a comment

JE-Chen commented Sep 18, 2024 •

edited

Loading

sungwy commented Sep 18, 2024

JE-Chen commented Sep 18, 2024 •

edited

Loading

sungwy commented Sep 18, 2024

JE-Chen commented Sep 18, 2024 •

edited

Loading

kevinjqliu commented Sep 18, 2024

JE-Chen commented Sep 18, 2024 •

edited

Loading

kevinjqliu commented Sep 18, 2024

JE-Chen commented Sep 20, 2024

kevinjqliu commented Sep 20, 2024

JE-Chen commented Sep 20, 2024 •

edited

Loading

kevinjqliu commented Sep 20, 2024

JE-Chen commented Sep 20, 2024

kevinjqliu left a comment

kevinjqliu commented Sep 20, 2024

Use ArrowScan.to_table to replace project_table #1180

Use ArrowScan.to_table to replace project_table #1180

Conversation

JE-Chen commented Sep 17, 2024 • edited Loading

sungwy commented Sep 17, 2024

JE-Chen commented Sep 17, 2024

kevinjqliu left a comment

Choose a reason for hiding this comment

JE-Chen commented Sep 18, 2024 • edited Loading

sungwy commented Sep 18, 2024

JE-Chen commented Sep 18, 2024 • edited Loading

steps

sungwy commented Sep 18, 2024

JE-Chen commented Sep 18, 2024 • edited Loading

kevinjqliu commented Sep 18, 2024

JE-Chen commented Sep 18, 2024 • edited Loading

kevinjqliu commented Sep 18, 2024

JE-Chen commented Sep 20, 2024

kevinjqliu commented Sep 20, 2024

JE-Chen commented Sep 20, 2024 • edited Loading

kevinjqliu commented Sep 20, 2024

JE-Chen commented Sep 20, 2024

kevinjqliu left a comment

Choose a reason for hiding this comment

kevinjqliu commented Sep 20, 2024

JE-Chen commented Sep 17, 2024 •

edited

Loading

JE-Chen commented Sep 18, 2024 •

edited

Loading

JE-Chen commented Sep 18, 2024 •

edited

Loading

JE-Chen commented Sep 18, 2024 •

edited

Loading

JE-Chen commented Sep 18, 2024 •

edited

Loading

JE-Chen commented Sep 20, 2024 •

edited

Loading