Skip to content

Commit

Permalink
[SPARK-33756][SQL] Make BytesToBytesMap's MapIterator idempotent
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?
Make MapIterator of BytesToBytesMap `hasNext` method idempotent

### Why are the changes needed?
The `hasNext` maybe called multiple times, if not guarded, second call of hasNext method after reaching the end of iterator will throw NoSuchElement exception.

### Does this PR introduce _any_ user-facing change?
NO.

### How was this patch tested?
Update a unit test to cover this case.

Closes #30728 from advancedxy/SPARK-33756.

Authored-by: Xianjin YE <advancedxy@gmail.com>
Signed-off-by: Sean Owen <srowen@gmail.com>
(cherry picked from commit 1339168)
Signed-off-by: Sean Owen <srowen@gmail.com>
  • Loading branch information
advancedxy authored and srowen committed Dec 20, 2020
1 parent 7e98ade commit 9b8c193
Show file tree
Hide file tree
Showing 2 changed files with 8 additions and 4 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -408,10 +408,12 @@ public void remove() {
}

private void handleFailedDelete() {
// remove the spill file from disk
File file = spillWriters.removeFirst().getFile();
if (file != null && file.exists() && !file.delete()) {
logger.error("Was unable to delete spill file {}", file.getAbsolutePath());
if (spillWriters.size() > 0) {
// remove the spill file from disk
File file = spillWriters.removeFirst().getFile();
if (file != null && file.exists() && !file.delete()) {
logger.error("Was unable to delete spill file {}", file.getAbsolutePath());
}
}
}
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -560,6 +560,8 @@ public void spillInIterator() throws IOException {
iter2.next();
}
assertFalse(iter2.hasNext());
// calls hasNext twice deliberately, make sure it's idempotent
assertFalse(iter2.hasNext());
} finally {
map.free();
for (File spillFile : spillFilesCreated) {
Expand Down

0 comments on commit 9b8c193

Please sign in to comment.