feat: support append data file and add e2e test #9
Conversation
ZENOTME
force-pushed
the
zj/tx_append
branch
from
b2fc5ff
to
4b3a6b9
Compare
|
Should we have another PR to rebase with the mainstream first? Or exclude them. |
|
Better to drop upstream commits. To rebase with upstream, we will need to force push or create a new branch |
ZENOTME
force-pushed
the
zj/tx_append
branch
from
4b3a6b9
to
dd9a186
Compare
crates/iceberg/src/transaction.rs
Outdated
| .filter(|entry| { | ||
| entry.has_added_files() | ||
| || entry.has_existing_files() | ||
| || entry.added_snapshot_id == snapshot_produce.snapshot_id |
There was a problem hiding this comment.
In which cases will this branch be triggered?
There was a problem hiding this comment.
Good catch. I ask this in the community and get the answer:
We've copied this from the Java code: https://github.com/apache/iceberg/blob/307593ffd99752b2d62cc91f4928285fc0c62b75/co[…]e/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java
I don't think this is used in PyIceberg today. In Java it is possible to add existing manifests to a commit, while in PyIceberg we only accept datafiles. This is on purpose since the ability to add manifests also comes with problems of its own.
ZENOTME
force-pushed
the
zj/tx_append
branch
from
dd9a186
to
16c4987
Compare
ZENOTME
force-pushed
the
zj/tx_append
branch
from
86fd4ac
to
2cd6703
Compare
ZENOTME
changed the base branch from
dev-rebase-main-20241030
to
dev-rebase-main-20241119
upstream pr: apache#349
This PR: