Adding Docker support for CaffeOnSpark #208
Conversation
Please note: Some of the Hadoop part extracted from "https://hub.docker.com/r/sequenceiq/hadoop-docker/~/dockerfile/" CaffeOnSpark is in directory /opt/CaffeOnSpark. Hadoop and Spark is set up correctly. COS working properly. Did MNIST example successfully.
Required: Proper SSH without password for Hadoop.
Required: To start hadoop dfs and yarn when starting container and to pass required environment variables.
|
CLA is valid! |
|
Docker support is a much needed feature which I was looking fwd to. I will
test it early next week. Thanks!
…On Thu, Dec 8, 2016 at 11:01 AM, Arun Das ***@***.***> wrote:
Hello,
Please test the docker file locally. Steps are documented in
docker/README.md.
The docker image is for CPU version of CaffeOnSpark. I will work on a GPU
version soon.
I checked the working twice on my local machine.
@anfeng <https://github.com/anfeng> , @mriduljain
<https://github.com/mriduljain> , Could you guys please let me know if
this is okay ? It was a bit tricky getting Hadoop to work on Docker. I had
to refer "https://hub.docker.com/r/sequenceiq/hadoop-docker/" to get it
working finally.
Also, I found that PATH variables were not set using Dockerfile. I had to
set them explicitly by calling a bootstrap.sh script when we start the
container for the first time (No need to use it while attaching to an
already existing container). bootstrap.sh also makes sure that hadoop dfs
and yarn starts with the container.
------------------------------
You can view, comment on, or merge this pull request online at:
#208
Commit Summary
- Create Dockerfile
- Create ssh_config
- Rename docker/standalone/cpu/ssh_config to
docker/standalone/cpu/config/ssh_config
- Create bootstrap.sh
- Create README.md
File Changes
- *A* docker/README.md
<https://github.com/yahoo/CaffeOnSpark/pull/208/files#diff-0> (19)
- *A* docker/standalone/cpu/Dockerfile
<https://github.com/yahoo/CaffeOnSpark/pull/208/files#diff-1> (155)
- *A* docker/standalone/cpu/config/bootstrap.sh
<https://github.com/yahoo/CaffeOnSpark/pull/208/files#diff-2> (27)
- *A* docker/standalone/cpu/config/ssh_config
<https://github.com/yahoo/CaffeOnSpark/pull/208/files#diff-3> (5)
Patch Links:
- https://github.com/yahoo/CaffeOnSpark/pull/208.patch
- https://github.com/yahoo/CaffeOnSpark/pull/208.diff
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#208>, or mute the thread
<https://github.com/notifications/unsubscribe-auth/ACCTVQQULYumBzvWq6ELjJ5AdiIj24Blks5rGFP1gaJpZM4LIKoR>
.
|
| MAINTAINER arun.das@my.utsa.edu | ||
|
|
||
| RUN apt-get update && apt-get install -y software-properties-common | ||
| RUN add-apt-repository ppa:openjdk-r/ppa |
There was a problem hiding this comment.
This line is repeated in 37 as well. Sorry about that. Didn't see it while uploading. Makes no errors, but redundant.
|
This is great idea! Thanks for contributing this! |
|
@javadba , were you able to get it working ? Could you please give it a try ? |
|
@arundasan91 Can you expand this PR with GPU support? |
|
@anfeng , sure can. I was waiting for a reply to make sure that this CPU version is enough. I will add a folder named I have a doubt on installing the correct version of CUDA based on the GPU's architecture and cuda compute. Will refer caffe's own way of doing it. |
removed redundant lines.
Adding GPU support for docker version of CaffeOnSpark
|
@anfeng , Please see the changes. A GPU version of the Dockerfile is added. I tested the same in an NVIDIA K80 environment. Works great. The only change to note here is to use Please let me know if there is any changes/additions required. |
|
@arundasan91 Good work. Can you add copyright notices to all new source files? @javadba Have you tried out this PR? |
|
@anfeng , Yes I will. I totally forgot to remove my name from the maintainer tag. Will do. |
|
@anfeng , Just to make it clear, these are the copyright lines right ? Shall I copy the same into every file in the PR ? Thanks. |
|
@anfeng I have not had a chance to try out the PR. Probably about two
weeks out due to some other fires. But definitely of interest - thanks.
2017-01-19 14:04 GMT-08:00 Arun Das <notifications@github.com>:
… @anfeng <https://github.com/anfeng> , Just to make it clear, these are
the copyright lines right ? Shall I copy the same into every file in the PR
?
# Copyright 2016 Yahoo Inc.
# Licensed under the terms of the Apache 2.0 license.
# Please see LICENSE file in the project root for terms.
#
# This file.......
Thanks.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#208 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAZEkdyxI6C_-Y0iSzQxv5AEYjvwl279ks5rT933gaJpZM4LIKoR>
.
|
|
@arundasan91 Yes, for all new files. |
added copyright
Added copyright
Added copyright
Added copyright
|
@anfeng , I added the copyright to every file. I added them to the Could you please take a look at the files and let me know. |
|
+1 @arundasan91 Excellent work. |
* 523b72e support build in mixed protobuf env * 2a83f82 Merge branch 'bluemind' into trlcaffe_merge |\ | * 16ec96f enhance makefile, no functional diff | * 27edffa fix a bug | * 379b673 add parameters for lmdbjni, spark, scala versions | * aa54efe fix a makefile bug which break the build | * 5f9f1ee build with new dli container image with protobuf and liblmdbjni version flexible | * 8356ad4 Merge pull request yahoo#5 from degaochu/bluemind | |\ | | * 316a1bc CaffeOnSpark copy weight file from Caffe is not real copy but move action . Change move file to copy file so that the file can inherit new diretory's acl attibutes | |/ * | 24c7ab2 support migrated BVLC caffe * | 8d26850 ankun's change about supporting NCCL enabled caffe * | 63ecedd switch multigpu to nccl * | f622620 update caffe-public submodule commit to trlmerge_yahoo2 * | 8c8b448 switch caffe-public to trlmerge_yahoo2 |/ * f8143cd pick up last bluemind caffe with fabric changes; switch caffe-public branch to "bluemind" * 2d9b8f0 Merge pull request yahoo#4 from sunweisw/bluemind_v0.3 |\ | * dc4d26f Fix hang issue with multiple gpu |/ * 2627390 Merge pull request yahoo#2 from fuzhiwen/dev |\ | * 224c7a8 S.138410: lmdb enhancement for CaffeOnSpark, migrated to bluemind v0.3 | * effdcff support build CoS with protobuf v3.2 |/ * ef28f91 formally moved to latest community version of caffeOnSpark 19df500 * 2ccaa1d Merge pull request yahoo#1 from sunweisw/merge-hist |\ | * 5d69279 Change for ppc64 env Disable test |/ * 19df500 Merge pull request yahoo#208 from arundasan91/patch-3
Hello,
Please test the docker file locally. Steps are documented in
docker/README.md.The docker image is for
CPUversion of CaffeOnSpark. I will work on a GPU version soon.I checked the working twice on my local machine.
@anfeng , @mriduljain , Could you guys please let me know if this is okay ? It was a bit tricky getting Hadoop to work on Docker. I had to refer "https://hub.docker.com/r/sequenceiq/hadoop-docker/" to get it working finally.
Also, I found that PATH variables were not set using Dockerfile. I had to set them explicitly by calling a
bootstrap.shscript when we start the container for the first time (No need to use it while attaching to an already existing container).bootstrap.shalso makes sure that hadoop dfs and yarn starts with the container.