spark-worker default_cmd Script not Working as Expected

[LuqmanSahaf]

I am using spark 1.0.0 docker images. It appears to me that the script default_cmd in spark-worker is not working as it should be working. This script calls prepare_spark $1 of /root/spark_files/configure_spark.sh. I have debugged it a lot. Even, I have called configure_spark.sh from spark-base image by using docker run -it.
The problem is that these script do not replace the __MASTER__ tag in core_site.xml in /root/hadoop_files/ with the argument provided. Instead, the worker expects the master to be master. That is, it is static.
Please, can someone help me out with this as I need it to create clusters on different machines. If I am not able to specify master like this, then I cannot create a cluster on different machines as the worker nodes will not know about the master. It works on single machine though, but that is because I have installed the docker-dns service.

[LuqmanSahaf]

I am sorry it is replacing the tag as it is supposed to. I was looking at /root/hadoop/core-site.xml instead of /etc/hadoop/core-site.xml. Once again, I am sorry. But the problem remains that why is the worker not contacting the master using its IP instead of the name 'master'.

[LuqmanSahaf]

I have created another issue and closing this one for the problem.
#44

[marcinjurek]

Hey I have the same problem. Were you able to figure it out?

[LuqmanSahaf]

@marcinjurek I had to edit the configure_spark.sh myself to let me use custom master.
Further, if you want to deploy on multiple machines you may have to use some kind of SD or packet forwarding. Look at coreos/flannel (I'm using it) and weave. Moreover, to use names you might have to edit the hosts file and not able to use docker-dns as is. You may want to use DNS as it's clean. You might want to my project LuqmanSahaf/etcd-spark to have an idea. I am not working on this project as now I'm setting Spark in completely different way. Hope this helps.

[LuqmanSahaf]

SDN*