Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test(duckdb): add test for chunk_size param in to_pyarrow_batches #10480

Merged
merged 2 commits into from
Nov 14, 2024
Merged

test(duckdb): add test for chunk_size param in to_pyarrow_batches #10480

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ad10b54
test(duckdb): add test for chunk_size param in to_pyarrow_batches
anjakefala Nov 12, 2024
32975b2
fixup: fix default backend, make sample data smaller
anjakefala Nov 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions ibis/backends/duckdb/tests/test_client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -420,3 +420,26 @@ def test_memtable_doesnt_leak(con, monkeypatch):
df = ibis.memtable({"a": [1, 2, 3]}, name=name).execute()
assert name not in con.list_tables()
assert len(df) == 3


def test_pyarrow_batches_chunk_size(con): # 10443
import numpy as np

t = ibis.memtable(
{
"id": np.arange(500000),
anjakefala marked this conversation as resolved.
Show resolved Hide resolved
"name": np.random.choice(["Alice", "Bob", "Carol", "Dave"], size=500000),
"age": np.random.randint(20, 70, size=500000),
}
)
batches = t.to_pyarrow_batches(chunk_size=100_000)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
batches = t.to_pyarrow_batches(chunk_size=100_000)
batches = con.to_pyarrow_batches(t, chunk_size=100_000)

You need to use con here, otherwise you'll implicitly set the default backend, which is what's currently causing CI to fail.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll absolutely change it! When does using the table implicitly set the default backend?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whenever you don't explicitly use a backend instance like con. Something has to do the compute, and if you haven't specified that thing explicitly by constructing a backend instance and then using it, or by calling ibis.set_backend(<some backend instance>) then Ibis is going to pick the default backend (DuckDB).

assert len(next(batches)) == 100_000
assert len(next(batches)) == 100_000

batches = t.to_pyarrow_batches(chunk_size=800)
anjakefala marked this conversation as resolved.
Show resolved Hide resolved
assert len(next(batches)) == 800
assert len(next(batches)) == 800

batches = t.to_pyarrow_batches(chunk_size=-1)
anjakefala marked this conversation as resolved.
Show resolved Hide resolved
with pytest.raises(TypeError):
next(batches)
Loading