-
Notifications
You must be signed in to change notification settings - Fork 29k
[SPARK-12481] [CORE] [STREAMING] [SQL] Remove usage of Hadoop deprecated APIs and reflection that supported 1.x #10446
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
Test build #48240 has finished for PR 10446 at commit
|
|
Test build #48243 has finished for PR 10446 at commit
|
|
Test build #2253 has finished for PR 10446 at commit
|
|
Test build #48366 has finished for PR 10446 at commit
|
|
Yea I think this one is fine. (I haven't looked at the detailed changes though) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this still needed? It seems you replaced the call with a direct call to the constructor in a lot of places.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yea, it's overridden in a mock object in OutputCommitCoordinatorSuite so I kept this
|
LGTM; there's at least one extra possible cleanup, but feel free to punt on that one. |
|
I'm pushing an update with those changes. I found a similar one in BTW here are the additional cleanups that could be made if Hadoop 2.6+ was assumed:
Taken together those don't add up to much advantage, but it's not nothing. The bigger reason would be less JAR hell to deal with and being able to access new APIs more freely. |
|
Test build #48475 has finished for PR 10446 at commit
|
|
Test build #2266 has finished for PR 10446 at commit
|
|
Merged to master |
… APIs and reflection that supported 1.x Remove use of deprecated Hadoop APIs now that 2.2+ is required Author: Sean Owen <sowen@cloudera.com> Closes #10446 from srowen/SPARK-12481.
Remove use of deprecated Hadoop APIs now that 2.2+ is required