Spark: Don't create empty partition replace operations #2960
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
Author
Member
Author
|
@binhnv Could please review as well |
When attempting to insert overwrite with an empty dataset we would previously throw an error. This patch causes spark to skip any no-op partition replacement operations.
RussellSpitzer
force-pushed
the
AllowEmptyReplace
branch
from
5d65f8e to
50d1aa5
Compare
rdblue
reviewed
Aug 11, 2021
| @Override | ||
| public void commit(WriterCommitMessage[] messages) { | ||
| Iterable<DataFile> files = files(messages); | ||
| if (Iterables.size(files) == 0) { |
Contributor
There was a problem hiding this comment.
What about using !files.hasNext instead? I'm not sure we want to assume that the iterable can be consumed multiple times. Plus there's no need to consume the entire iterable just to check whether it is empty.
rdblue
approved these changes
Aug 11, 2021
Contributor
rdblue
left a comment
rdblue
left a comment
There was a problem hiding this comment.
I left a minor comment, but this looks good to me.
RussellSpitzer
force-pushed
the
AllowEmptyReplace
branch
from
79e63aa to
975d811
Compare
|
Thank you for fixing this. The change looks good to me and it is also consistent with Hive's behavior. |
Member
Author
hankfanchiu
added a commit
to hankfanchiu/iceberg
that referenced
this pull request
Aug 27, 2021
chenjunjiedada
pushed a commit
to chenjunjiedada/incubator-iceberg
that referenced
this pull request
Oct 20, 2021
Merge remote-tracking branch 'upstream/merge-master-20210816' into master ## 该MR主要解决什么？ merge upstream/master，引入最近的一些bugFix和优化 ## 该MR的修改是什么？ 核心关注PR： > Predicate PushDown 支持，https://github.com/apache/iceberg/pull/2358， https://github.com/apache/iceberg/pull/2926， https://github.com/apache/iceberg/pull/2777/files > Spark场景写入空dataset 报错问题，直接skip掉即可， apache#2960 > Flink UI补充uidPrefix到operator方便跟踪多个iceberg sink任务， apache#288 > Spark 修复nested Struct Pruning问题， apache#2877 > 可以使用Table Properties指定创建v2 format表，apache#2887 > 补充SortRewriteStrategy框架，逐步支持不同rewrite策略， apache#2609 （WIP：apache#2829） > Spark 为catalog配置hadoop属性支持， apache#2792 > Spark 针对timestamps without timezone读写支持， apache#2757 > Spark MicroBatch支持配置属性skip delete snapshots， apache#2752 > Spark V2 RewriteDatafilesAction 支持 > Core: Add validation for row-level deletes with rewrites, apache#2865 > schema time travel 功能相关，补充schema-id， Core: add schema id to snapshot > Spark Extension支持identifier fields操作， apache#2560 > Parquet: Update to 1.12.0, apache#2441 > Hive: Vectorized ORC reads for Hive, apache#2613 > Spark: Add an action to remove all referenced files, apache#2415 ## 该MR是如何测试的？ UT
This was referenced Feb 15, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When attempting to insert overwrite with an empty dataset
we would previously throw an error. This patch causes spark
to skip any no-op partition replacement operations.