ODPi Runtime v3 Outline

Abstract
Included Projects
- Apache Hadoop® 3.0, including all maintenance releases.
- Apache Hadoop® 3.0, compatible filesystems (HCFS).
- Apache Hive™ 3.0
- Apache Spark™ 2.2
- Apache Ranger™ 1.0
- Apache Atlas™ 1.0.0
Objective
Technical Context
Specification Format
Upgrade Hadoop v2.x.y to Hadoop v3.0.0 Considerations
- Minimum required Java version increased from Java 7 to Java 8
- YARN Timeline Service v.2 (should be used only in a test capacity)
- Shell script rewrite
New Features You Might Find Useful
- Erasure coding
- Support for more than 2 NameNodes.
- Support for Opportunistic Containers and Distributed Scheduling
- MapReduce task-level native optimization
- Shaded client jars
- Ozone - S3-compatible API storage (Hadoop v3.2)
- GPUs supported as a first-class resource type (Hadoop v3.1)
- YARN support for Docker containers (Hadoop v3.1)
How To Upgrade from Hadoop v2 to v3
Challenges
Deployment Best Practices
Where to get the Binaries
Build Specifications
- Version Specifications
- Patch Specifications
- Minimum Native build specifications
  - Common
  - HDFS
  - YARN
  - MapReduce
Runtime Environment for Application Code
- Minimum Versions
- Environment Variables
- Compliance
  - Hadoop Compliance
  - HCFS (Hadoop-compatible filesystem) Compliance
  - Hive Compliance
- Compatibility
- Glossary
Appendix A, Hadoop Configuration Values Not Shared
Appendix B, Hive Configuration Values to be White Listed

Provide feedback