installing a framework after teardown

[sadovnikov]

Using Marathon I removed ES Scheduler, but this left Executors running. In order to remove them I teared down the framework by curl -X POST -d 'frameworkId=XXXXXXXX-b036-4cb7-af53-4c837dc9521d-0002' http://${MASTER_IP}:5050/master/teardown. This successfully removed all the framework tasks - executors.

However now Mesos Cluster rejects my attempts to re-install the framework.

I0115 12:54:57.916470 28856 sched.cpp:1024] Got error 'Framework has been removed'
I0115 12:54:57.916509 28856 sched.cpp:1805] Asked to abort the driver
I0115 12:54:57.916824 28856 sched.cpp:1070] Aborting framework '8ca5c18f-b036-4cb7-af53-4c837dc9521d-0001'

Below I copy an answer from user@mesos.apache.org. Is it possible the scheduler gets the old ID from Zookeeper? What changes should we introduce to enable re-installation of the framework?

[sadovnikov]

from @massenz on user@mesos.apache.org

I'm not clear what you mean by "re-install the same framework" - do you mean, just restarting the binary?
If so, as Vinod pointed out, you should re-register with a SUBSCRIBE message and obtain a new FrameworkId in the response.

And, yes, the name can stay perfectly the same (in fact, you can have several frameworks with the same name - but different IDs - connect to the same Master).

Are you using the C++ API or the new HTTP API?

If the latter, please have a look at the example here[0] for how to "terminate" a framework and then reconnect it.
(in particular, see the sections "Registering a Framework" and "Terminating a Framework").

If the former, see [1] where I set the name in the FrameworkInfo (and that one stays the same across runs) but not the ID (that one gets returned in the registered()[2] method and can be used, if necessary elsewhere in the code; for example, when accepting offers; but should otherwise stay unique to the FW).

There are many (better!) examples around of frameworks also in the "Examples" folder in the Mesos source code[3]; you may want to take a look there too.

[0] https://github.com/massenz/zk-mesos/blob/develop/notebooks/Demo-API.ipynb
[1] https://github.com/massenz/mongo_fw/blob/develop/src/mongo_scheduler.cpp#L194
[2] https://github.com/massenz/mongo_fw/blob/develop/src/mongo_scheduler.cpp#L70
[3] https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=tree;f=src/examples

[sadovnikov]

I believe we need to provide ways to achieve the following:

• removal of all running ES nodes; tearing down the framework is one of the options;
• if somebody teared down the framework, Scheduler should be able to reconnect with Mesos Master. Keep in mind, Scheduler could be already stopped when the framework is teared down, therefore we can not reply on ability to remove records from ZK

[frankscholten]

@philwinder @sadovnikov We should check if there is a callback to the scheduler when tearing down the framework. If so we can use that to remove records from ZK.

[massenz]

(@sadovnikov - thanks for attributing the quote)

Guys - if I can be of any help, feel free to ping me directly (m.massenzio (at) gmail com) - github notifications unfortunately are somewhat lacking and I saw this one with a bit of delay.

cheers,
Marco.

[philwinder]

@sadovnikov The only possible place where we may get a callback is in:

/**
   * Invoked when the scheduler becomes "disconnected" from the master
   * (e.g., the master fails and another is taking over).
   *
   * @param driver  The driver that was used to run this scheduler.
   *
   * @see SchedulerDriver
   */
  void disconnected(SchedulerDriver driver);

Can you check to see whether you can see a log message in the ES scheduler saying "disconnected"?
I remember a discussion with one of the German Marathon devs that we see all the time saying that marathon has to manually kill the scheduler application when someone tries to kill the framework.

[tpolekhin]

The scheduler itself can fail and restart in marathon.
If we gonna remove data from ZK on scheduler's termination will the new spawned one be able to connect to existing running es-executors?

[philwinder]

@tymofii-polekhin @sadovnikov is specifically talking about uninstalling, i.e. the complete removal of a framework. Not failover.

[sadovnikov]

Callbacks are good, however it could be there is nothing to receive them. I think reuse of framework ID from ZooKeeper is a good idea. However if this does not work, the scheduler, if given permitting parameter, should reconnect to Master and register new ID

[philwinder]

That's a noble cause, but is difficult with the API. The Mesos scheduler API works like this:
Connect to the master
Send a framework registration request
Receive a callback saying you're registered.

Note the callback. If the master flips out and decides not to register you (because the framework has already been shutdown, for example) then you won't receive a callback. The framework will just sit there waiting. So using their API's, there is no way of knowing whether it worked.

The only way to achieve this is to somehow interrogate mesos to see if that framework has already terminated, before sending the registration request. I suspect this is possible, but I would have to look into where to get the information from.

Also note that this is nothing to do with zookeeper. It's all Mesos. ZK is just a datastore.

[sadovnikov]

Thanks for clarification!

[massenz]

I'm not sure why you may want to preserver Framework IDs across restarts (or, even more so, re-installs) but as @philwinder correctly pointed out, that's at odds with how Mesos uses them - even assuming you can hack your way round it (and I doubt it) you are likely to encounter weird (and hard-to-debug) failures.

I would see the ID as a "throwaway" (mostly opaque to your framework) internal ledger-keeping for Mesos to worry about: a bit like the barcode on the ticket - it matters a lot to the organizers and gatekeepers at the game, but you don't much care about it, so long as your seat is reserved (and vacant!) for you.

If you can explain why preserving it matter, maybe I can help finding a different way to accomplish the same?

[philwinder]

Ok, the real solution here is to correctly interpret the various sigkill/sigterm signals sent to the scheduler. We need to check, but when you call teardown, I assume it sends a sigterm to the scheduler. Also, when a user wants to shut down all the executors, they can send a sigterm. The code should interpret this, kill all the executors, remove the state from ZK and quit. On a sigkill however, this should be treated as a failure and the executors should stay alive and the zk state not removed.

This way, if you teardown the framework, it will also remove all executors and the zk state so a new instance can correctly start.

@mwl Has just implemented something similar in the Logstash framework.

[philwinder]

Tasks:

• Ensure sh wrappers pass on signals to java
• Write code to interpret signals
• If sigterm, kill all executors. Delete all ZK state.
• Else do nothing.
• Double check that marathon sends a sigterm when terminating a framework
• Double check that mesos sends a sigterm upon teardown.
• Test that users are able to reinstall the framework after teardown

[mwl]

It's a bit different for Logstash. We rely heavily on Spring, so it's just a matter of using a @PreDestroy annotation on a method.
But I believe it's just at matter of adding a shutdown hook somewhere?

[mwl]

Ensure sh wrappers pass on signals to java

Honestly, the easiest is to launch java directly from Dockerfile. Start script just complicates things.

[philwinder]

Fixed in #509.

From the sh script, starting java with exec passes the signals to Java. Had to implement zk removal code.