No tasks started on deploying elasticsearch-scheduler over mesos

[pradeepchhetri]

Hello,

I have a mesos + marathon cluster with mesos-slaves of type t2.medium ec2 instance. I deployed elasticsearch-scheduler over marathon according to the documentation but with docker.network value as BRIDGE instead of HOST. But i didn't find any elasticsearch-executor starting. I am unable to figure out the reason of elasticsearch-executor not coming up automatically. Can somone help me in debugging the issue.

Thanks.

[philwinder]

Hi Pradeep,
The most likely cause is that you are using the instructions from the master branch, but we haven't released the 0.3.0 docker images yet. This version alters the CLI, so old versions won't work. Please look at the readme.md from the 0.2.1 tagged version, or wait a few days until we release 0.3.0.

On the other issue, the HOST mode is required. BRIDGE mode is untested and probably won't work.

Try again with these points in mind, and if you still have problems, get back in touch. If you do have problems, look at the scheduler and/or executor logs and paste them here. They are accessible via the Mesos gui.

Thanks, Phil

[pradeepchhetri]

@philwinder Ahh i see, maybe you can help the newbies by making "0.2.1" as the default branch in github settings so that whosoever visits the github repo is automatically redirected to "0.2.1" branch. I will try it out.

Thank you for the quick reply Phil :)

[pradeepchhetri]

@philwinder I am able to run the elasticsearch over mesos cluster but i am not able to figure out the elasticsearch endpoint. I can see the es-scheduler and es-executor running in mesos dashboard.

[frankscholten]

Can you access the GUI on port 8080 on the scheduler's machine? Go to the tasks tab and there you can navigate to the elasticsearch endpoints.

[pradeepchhetri]

@frankscholten Thank you for your quick reply. I started es-scheduler with --numHardwareNodes 2 but i can see only one executor as up. Is it to do something with the resource. Moreover the tasks tab shows 4 entries, out of which only one works.

# docker ps
CONTAINER ID        IMAGE                                                               COMMAND                CREATED             STATUS              PORTS                     NAMES
86846c8af36f        mesos/elasticsearch-executor                                        "/tmp/start-executor   56 minutes ago      Up 56 minutes                                 mesos-8e69000f-3c04-40e0-a528-ae83b6bfa2b4
11fbe0a146ce        mesos/elasticsearch-scheduler                                       "/tmp/start-schedule   57 minutes ago      Up 57 minutes                                 mesos-8cce89e8-9c5b-4af3-b4ae-d81307f551b6

[frankscholten]

The GUI showing 4 entries when there is 2 tasks is a known issue, #206, see https://github.com/mesos/elasticsearch#known-issues for more information.

You were able to search or index on the endpoint that is available?

[pradeepchhetri]

@frankscholten Ahh, i see. Yes i am able to index data and search on top of the endpoint that is available. I have few questions for you:

1. Since the elasticsearch-executor are running on random mesos-slave and random port, is there any way to uniquely identify them using mesos-dns or some other service discovery tool.
2. I am sending the data directly to elasticsearch using rsyslog-omelasticsearch plugin. Currently, If the executor goes down, i need to update all the rsyslog config on all the client. So i definitely need some kind of service discovery for this. Do you have any in-built way to tackle this.

[pradeepchhetri]

@frankscholten is there a way i can set an environment variable inside elasticsearch-executor. I can do that for elasticsearch-scheduler since we are posting that json to marathon. This way we can easily integrate it with consul (tool for service discovery) using registrator.

[frankscholten]

1. The framework adds a DiscoveryInfo to every task with port numbers that are used by the executors. This information appears at http://master:5050/state.json. However it is not yet possible to configure visibility per port range, see https://issues.apache.org/jira/browse/MESOS-2960
2. Even though we provide DiscoveryInfo the framework does not support service discovery out of the box. Currently, the framework does not provide a way to add environment variables to the executor. Internally we do add environment variables to the CommandInfo in the executor. Have a look at TaskInfoFactory and the method newCommandInfo where we add the variables.

[pradeepchhetri]

@frankscholten Thank you for the quick replies. I will try playing with it and see if i can do something. Can I make sure that all elasticsearch nodes start at a particular port instead of random port. Is there any configuration piece which i can change for that ?

[pradeepchhetri]

Did we push new version of docker images because i see the image creation time got updated.

[frankscholten]

We have released 0.3.0 last friday