-
Notifications
You must be signed in to change notification settings - Fork 29k
[SPARK-12130] Replace shuffleManagerClass with shortShuffleMgrNames in ExternalShuffleBlockResolver #10131
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
looks OK, but why not add a field in ShuffleManager itself that sets this? So you can just do shuffleManager.name or something here
|
@andrewor14 Now I add shortShuffleMgrName to ShuffleManager for getting its short Name. How do you think? Thanks. |
|
Test build #47197 has finished for PR 10131 at commit
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would make these vals since we don't ever expect them to change within a shuffle manager
|
Minor comment but this looks good otherwise. |
|
@andrewor14 thanks. I think now it's ok. |
|
Test build #47710 has finished for PR 10131 at commit
|
|
Test build #47711 has finished for PR 10131 at commit
|
|
Test build #47716 has finished for PR 10131 at commit
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
just call this shortName
|
LGTM, merging into master. I'll fix the comment on merge. |
|
@andrewor14 OK, Thanks. |
Replace shuffleManagerClassName with shortShuffleMgrName is to reduce time of string's comparison. and put sort's comparison on the front. cc @JoshRosen @andrewor14