Big Data Processing with Docker and Hadoop not working

[jjkoehorst]

Which lab is it that you're having issues with?

https://github.com/docker/labs/blob/master/developer-tools/java/chapters/ch11-bigdata.adoc

Description

maven clean install fails as well as the docker-compose

Steps to reproduce the issue, if relevant:

1. git clone https://github.com/fabianenardon/hadoop-docker-demo
2. cd sample
3. mvn clean install -Papp-docker-image

Describe the results you received:

➜  sample git:(master) mvn clean install -Papp-docker-image
[INFO] Scanning for projects...
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] Building docker-hadoop-example 1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ docker-hadoop-example ---
[INFO] Deleting /Users/xxx/GitLab/hadoop-docker-demo/sample/target
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ docker-hadoop-example ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] 
[INFO] --- maven-compiler-plugin:2.5.1:compile (default-compile) @ docker-hadoop-example ---
[INFO] Compiling 1 source file to /Users/xxx/GitLab/hadoop-docker-demo/sample/target/classes
[INFO] 
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ docker-hadoop-example ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 4 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.5.1:testCompile (default-testCompile) @ docker-hadoop-example ---
[INFO] Compiling 2 source files to /Users/xxx/GitLab/hadoop-docker-demo/sample/target/test-classes
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ docker-hadoop-example ---
[INFO] Surefire report directory: /Users/xxx/GitLab/hadoop-docker-demo/sample/target/surefire-reports

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running org.examples.hadoop.AppTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.047 sec

Results :

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

[INFO] 
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ docker-hadoop-example ---
[INFO] Building jar: /Users/xxx/GitLab/hadoop-docker-demo/sample/target/docker-hadoop-example-1.0-SNAPSHOT.jar
[INFO] 
[INFO] --- maven-assembly-plugin:2.3:single (make-mr) @ docker-hadoop-example ---
[INFO] Reading assembly descriptor: /Users/xxx/GitLab/hadoop-docker-demo/sample/src/main/resources/assembly/hadoop-mr.xml
[INFO] Building jar: /Users/xxx/GitLab/hadoop-docker-demo/sample/target/docker-hadoop-example-1.0-SNAPSHOT-mr.jar
[INFO] 
[INFO] --- docker-maven-plugin:0.20.0:build (docker-build) @ docker-hadoop-example ---
[WARNING] The assembly descriptor contains a filesystem-root relative reference, which is not cross platform compatible /jar
[WARNING] The assembly descriptor contains a filesystem-root relative reference, which is not cross platform compatible /test-data
[INFO] Copying files to /Users/xxx/GitLab/hadoop-docker-demo/sample/target/docker/docker-hadoop-example/build/maven
[INFO] Building tar: /Users/xxx/GitLab/hadoop-docker-demo/sample/target/docker/docker-hadoop-example/tmp/docker-build.tar
[INFO] DOCKER> docker-build.tar: Created [docker-hadoop-example] in 426 milliseconds
[INFO] DOCKER> [docker-hadoop-example]: Built image sha256:f8a3b
[INFO] DOCKER> [docker-hadoop-example]: Removed old image sha256:27f90
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 9.719 s
[INFO] Finished at: 2018-01-07T08:05:12+01:00
[INFO] Final Memory: 41M/741M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal io.fabric8:docker-maven-plugin:0.20.0:build (docker-build) on project docker-hadoop-example: Execution docker-build of goal io.fabric8:docker-maven-plugin:0.20.0:build failed: JSONObject["auth"] not found. -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
➜  sample git:(master) 

And for the docker image

➜  docker git:(master) docker-compose up -d
Pulling mongo (library/mongo:3.2.12)...
3.2.12: Pulling from library/mongo
47e616392c23: Pull complete
b8dc044ce904: Pull complete
4ccb5ca0e379: Pull complete
3b1e4f984392: Pull complete
8a6016a58337: Pull complete
a21527fa8188: Pull complete
44adcaad2c76: Pull complete
ac4de227c77e: Pull complete
d2f665d49ba4: Pull complete
766e94e08ac1: Pull complete
b89f35cc5a28: Pull complete
Digest: sha256:af40f41f870c15e41d48bf2433134e0577f97d5ebbf55c4a2214ed499026a08e
Status: Downloaded newer image for mongo:3.2.12
Pulling namenode (tailtarget/hadoop:2.7.2)...
2.7.2: Pulling from tailtarget/hadoop
df559435c037: Pull complete
259a0ed5f842: Pull complete
48faacd68e4d: Pull complete
7c4da29bc4b0: Pull complete
6085af4413d2: Pull complete
0806b3320a96: Pull complete
d09d0ad85826: Pull complete
133f0a6cda1c: Pull complete
0d1096926a24: Pull complete
eaee71f9f8a3: Pull complete
74a40df51ea7: Pull complete
Digest: sha256:462f78a855186a8c582a7d33f5aa40158dc21600838997dd9bdface262364f38
Status: Downloaded newer image for tailtarget/hadoop:2.7.2
Pulling docker-hadoop-example (docker-hadoop-example:latest)...
ERROR: pull access denied for docker-hadoop-example, repository does not exist or may require 'docker login'

Output of docker version:

Client:
 Version:      17.09.1-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   19e2cf6
 Built:        Thu Dec  7 22:22:25 2017
 OS/Arch:      darwin/amd64

Server:
 Version:      17.09.1-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   19e2cf6
 Built:        Thu Dec  7 22:28:28 2017
 OS/Arch:      linux/amd64
 Experimental: true

Output of docker info:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 2
Server Version: 17.09.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.49-moby
Operating System: Alpine Linux v3.5
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 9.498GiB
Name: moby
ID: XFHH:45XY:IWMT:Z3EL:4T5K:FTR4:K5I7:KPW5:CJY5:GSYU:YXG2:43UB
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 18
 Goroutines: 29
 System Time: 2018-01-07T07:15:18.975401986Z
 EventsListeners: 1
No Proxy: *.local, 169.254/16
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, Docker for Mac, Docker for Windows, VirtualBox, physical, etc.):

Using docker for mac and java 1.8

java version "1.8.0_141"
Java(TM) SE Runtime Environment (build 1.8.0_141-b15)
Java HotSpot(TM) 64-Bit Server VM (build 25.141-b15, mixed mode)

[fabianenardon]

Maybe related to fabric8io/docker-maven-plugin#731 ? Can you check your ~/.docker/config.json file?

The docker-compose step does not work because the image is generated in the mvn clean install step. If it fails, the other will fail either.

[jjkoehorst]

Yes, moving the config.json to config_backup.json it compiles and runs smoothly... Still the bug remains but thanks for the work around!