Apache Hadoop 2.7.4 Docker image

Docker image for a pseudo-distributed Hadoop 2.7.4 installation with JDK 1.8u144.

Forked from: sequenceiq/hadoop-docker, original README preserved below.

Everything is up on docker hub at gvacaliuc/hadoop-docker. Pushes to master update this image.

building

$ docker build -t gvacaliuc/hadoop-docker .

pulling

Pulling from :latest should be fine as I'm not going to be updating this repo that often, but you can optionally specify :2.7.4 as the image version if you'd like to be sure the image will work.

$ docker pull gvacaliuc/hadoop-docker:latest

running

$ docker run -it gvacaliuc/hadoop-docker:latest /etc/bootstrap.sh -bash

Optionally provide a hostname:

$ docker run -h awesome-hostname -it gvacaliuc/hadoop-docker:latest /etc/bootstrap.sh -bash

what happens when I run the /etc/bootstrap.sh script?

1. some hadoop environment variables get set
2. the hostname of the docker image gets dumped into some hadoop config
3. sshd (ssh daemon) gets started, listening on port 2212
4. hdfs gets started
5. depending on the flag to /etc/bootstrap
   • if -d, the script launches as a "daemon" and sits in an infinite while loop
   • if -bash the script lands you in a bash shell in the container

connecting to the running container

If you launched the container as a daemon, i.e. -d flag to the bootstrap script, then you need a way to actually connect to the container. If you named the container when you launched it, you can launch a bash shell with

$ docker exec -it awesomeContainerName bash

If you didn't name it, just use docker ps to get the container name / id and use one of those. If this is the last container you launched, you can also just use

$ docker exec -it $(docker ps -lq) bash

which gets the latest (-l) container id only (-q).

using docker compose

If you have docker compose installed, you can use the included docker-compose.yml to get started. It sets the hostname and mounts the directory it was launched from as /launchdir on the container. Example usage:

$ docker-compose run hadoop

BEGIN OLD README

---

A few weeks ago we released an Apache Hadoop 2.3 Docker image - this quickly become the most popular Hadoop image in the Docker registry.

Following the success of our previous Hadoop Docker images, the feedback and feature requests we received, we aligned with the Hadoop release cycle, so we have released an Apache Hadoop 2.7.1 Docker image - same as the previous version, it's available as a trusted and automated build on the official Docker registry.

FYI: All the former Hadoop releases (2.3, 2.4.0, 2.4.1, 2.5.0, 2.5.1, 2.5.2, 2.6.0) are available in the GitHub branches or our Docker Registry - check the tags.

Build the image

If you'd like to try directly from the Dockerfile you can build the image as:

docker build  -t sequenceiq/hadoop-docker:2.7.1 .

Pull the image

The image is also released as an official Docker image from Docker's automated build repository - you can always pull or refer the image when launching containers.

docker pull sequenceiq/hadoop-docker:2.7.1

Start a container

In order to use the Docker image you have just build or pulled use:

Make sure that SELinux is disabled on the host. If you are using boot2docker you don't need to do anything.

docker run -it sequenceiq/hadoop-docker:2.7.1 /etc/bootstrap.sh -bash

Testing

You can run one of the stock examples:

cd $HADOOP_PREFIX
# run the mapreduce
bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar grep input output 'dfs[a-z.]+'

# check the output
bin/hdfs dfs -cat output/*

Hadoop native libraries, build, Bintray, etc

The Hadoop build process is no easy task - requires lots of libraries and their right version, protobuf, etc and takes some time - we have simplified all these, made the build and released a 64b version of Hadoop nativelibs on this Bintray repo. Enjoy.

Automate everything

As we have mentioned previousely, a Docker file was created and released in the official Docker repository