-
Notifications
You must be signed in to change notification settings - Fork 3.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
HBASE-27590 Change Iterable to List in SnapshotFileCache #4995
Conversation
This comment was marked as outdated.
This comment was marked as outdated.
S3 is not the only HFile storage we support, so I think we should also consider the performance impact on HDFS, at least. And better start a discussion thread on the dev list to gain more feedbacks. |
This comment was marked as outdated.
This comment was marked as outdated.
Let me do a quick test on HDFS if it impacts the performance and after I'll send a mail to the dev list. |
This comment was marked as outdated.
This comment was marked as outdated.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
agreed with the point that Duo said about other storage, and thanks for testing HDFS and mentioned the performance difference.
let's see how the discussion goes.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh i see. It's because guava's Iteratorable APIs are lazy:
https://guava.dev/releases/21.0/api/docs/com/google/common/collect/Iterables.html
Performance notes: Unless otherwise noted, all of the iterables produced in this class are lazy, which means that their iterators only advance the backing iteration when absolutely necessary.
...server/src/main/java/org/apache/hadoop/hbase/replication/master/ReplicationHFileCleaner.java
Outdated
Show resolved
Hide resolved
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
So the actual problem here is lazy evaluation which will delay the actual io operation under lock? Let me take a look at the PR about why lazy evaluation produce bad result here... |
As replied on the dev mailing list, I do not think we need to do a big refactoring, just change the way we call Thanks. |
2ced999
to
49e5930
Compare
Thanks for the suggestion @Apache9! I've tested with this minor change and the improvement was the same. |
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotHFileCleaner.java
Show resolved
Hide resolved
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
🎊 +1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
@Apache9 Please take a look. Thanks. |
Signed-off-by: Duo Zhang <zhangduo@apache.org> (cherry picked from commit d2c5af1)
Signed-off-by: Duo Zhang <zhangduo@apache.org> (cherry picked from commit d2c5af1)
Signed-off-by: Duo Zhang <zhangduo@apache.org> (cherry picked from commit d2c5af1)
No description provided.