Python package size has ballooned dramatically

[gorilla-seb]

From 0.4.0 to 0.5.0 the size of pyiceberg-core Python packages has increased dramatically.

Python wheels for 0.4.0: between ~ 0.7 and 1.5 MiB

Python wheels for 0.5.1: 25 to 55MiB

This is reflected in the installed package (example from OSX, but Linux is similar):

% du -sh .venv/lib/python3.12/site-packages/pyiceberg_core/*
4.0K    .venv/lib/python3.12/site-packages/pyiceberg_core/__init__.py
4.0M    .venv/lib/python3.12/site-packages/pyiceberg_core/pyiceberg_core_rust.abi3.so

% poetry update pyiceberg-core
Package operations: 0 installs, 1 update, 0 removals

  - Updating pyiceberg-core (0.4.0 -> 0.5.1)

% du -sh .venv/lib/python3.12/site-packages/pyiceberg_core/*
4.0K    .venv/lib/python3.12/site-packages/pyiceberg_core/__init__.py
162M    .venv/lib/python3.12/site-packages/pyiceberg_core/pyiceberg_core_rust.abi3.so

In increase by over 4000% between minor versions seems like something is wrong with the build. Moreover, packages this large are borderline unusable on a service like AWS lambda where total size is limited to 250MiB.

[kevinjqliu]

hey @gorilla-seb i believe this is the result of #1324 when we added the datafusion table provider to the python binding.

In increase by over 4000% between minor versions seems like something is wrong with the build

this is because pyiceberg_core only packaged the transform functions before,

iceberg-rust/bindings/python/src/lib.rs

Line 28 in f1e79c0

transform::register_module(py, m)?;

. And those are pretty small

Moreover, packages this large are borderline unusable on a service like AWS lambda where total size is limited to 250MiB.

pyiceberg_core is mainly used for the pyiceberg library since it doesnt have any standalone features.

[kevinjqliu]

Closing this for now

[kevinjqliu]

btw we reduce the size of the wheels significantly in #1839