Skip to content

Conversation

@srowen
Copy link
Member

@srowen srowen commented Dec 23, 2015

Remove use of deprecated Hadoop APIs now that 2.2+ is required

@SparkQA
Copy link

SparkQA commented Dec 23, 2015

Test build #48240 has finished for PR 10446 at commit 2c9f628.

  • This patch fails MiMa tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Dec 23, 2015

Test build #48243 has finished for PR 10446 at commit 9864b68.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Dec 23, 2015

Test build #2253 has finished for PR 10446 at commit 9864b68.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Dec 28, 2015

Test build #48366 has finished for PR 10446 at commit 1fffa8e.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@srowen
Copy link
Member Author

srowen commented Dec 29, 2015

Pinging folks like @vanzin @pwendell @rxin @sarutak
I don't think this one is controversial, as it just follows from removing older Hadoop support and cleans up the build. Still, it's not a small change, so wouldn't mind any other set of eyes on it.

@rxin
Copy link
Contributor

rxin commented Dec 29, 2015

Yea I think this one is fine.

(I haven't looked at the detailed changes though)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this still needed? It seems you replaced the call with a direct call to the constructor in a lot of places.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, it's overridden in a mock object in OutputCommitCoordinatorSuite so I kept this

@vanzin
Copy link
Contributor

vanzin commented Dec 30, 2015

LGTM; there's at least one extra possible cleanup, but feel free to punt on that one.

@srowen
Copy link
Member Author

srowen commented Dec 30, 2015

I'm pushing an update with those changes. I found a similar one in FileBasedWriteAheadLogWriter and one more in FsHistoryProvider regarding checking for safe mode.

BTW here are the additional cleanups that could be made if Hadoop 2.6+ was assumed:

  • SparkHadoopUtil: getFileSystemThreadStatistics, getFSBytesReadOnThreadCallback and getFSBytesWrittenOnThreadCallback only actually work on Hadoop 2.5+
  • Hadoop RDDs could always directly access SplitLocationInfo, InputSplitWithLocationInfo introduced in 2.5
  • YARN integration tests could directly access tag-related APIs like ApplicationContext getApplicationTags from 2.4 (?)
  • AM could cleanly handle ApplicationAttemptNotFoundException

Taken together those don't add up to much advantage, but it's not nothing. The bigger reason would be less JAR hell to deal with and being able to access new APIs more freely.

@SparkQA
Copy link

SparkQA commented Dec 30, 2015

Test build #48475 has finished for PR 10446 at commit 5e3318a.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Dec 30, 2015

Test build #2266 has finished for PR 10446 at commit 5e3318a.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@srowen
Copy link
Member Author

srowen commented Jan 2, 2016

Merged to master

@srowen srowen closed this Jan 2, 2016
@srowen srowen deleted the SPARK-12481 branch January 2, 2016 13:15
asfgit pushed a commit that referenced this pull request Jan 2, 2016
… APIs and reflection that supported 1.x

Remove use of deprecated Hadoop APIs now that 2.2+ is required

Author: Sean Owen <sowen@cloudera.com>

Closes #10446 from srowen/SPARK-12481.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants