fix: Iceberg fixes for reading table metadata #2810
Conversation
vrongmeal
force-pushed
the
vrongmeal/iceberg
branch
from
3ca6c31 to
56a21c0
Compare
vrongmeal
changed the title
|
What more do we need here? (I think there's some more test coverage that we could use here?) |
vrongmeal
force-pushed
the
vrongmeal/iceberg
branch
from
ca9eba9 to
b4f7e5e
Compare
Added test coverage. There are issues though. If we overwrite the existing data (say running the |
testdata/generate_pyiceberg.py
Outdated
| How to run this script: | ||
| ====================== | ||
|
|
||
| $ python3 -m venv venv | ||
| $ source venv/bin/activate | ||
| $ pip install "pyiceberg[pyarrow,sql-sqlite]" | ||
| $ pip install botocore | ||
| $ python ./testdata/generate_pyiceberg.py |
There was a problem hiding this comment.
I think the data set ends up being pretty big, and while I don't think it really matters, I think it'd be good to avoid just committing lots of test data to the repo just cause, when we have other options (e.g. uploading the data to the test bucket, or generating as part of a fixture in one of the pytests).
There was a problem hiding this comment.
Yeah, thinking the same thing.
Signed-off-by: Vaibhav <vrongmeal@gmail.com>
Signed-off-by: Vaibhav <vrongmeal@gmail.com>
Signed-off-by: Vaibhav <vrongmeal@gmail.com>
Signed-off-by: Vaibhav <vrongmeal@gmail.com>
vrongmeal
force-pushed
the
vrongmeal/iceberg
branch
from
5e411ec to
c7bdfaa
Compare
Signed-off-by: Vaibhav <vrongmeal@gmail.com>
|
@tychoish made the required changes for pytest |
Signed-off-by: Vaibhav <vrongmeal@gmail.com>
There was a problem hiding this comment.
45mb feels big for a test file.
There was a problem hiding this comment.
Yeah, I'll upload it on GCS. We have a script to handle big data files.
There was a problem hiding this comment.
It was a simple HTTP url, added it into the fixture to download the data.
Signed-off-by: Vaibhav <vrongmeal@gmail.com>
Fixes #2603