Workaround required for unreachable resident tasks

[timcharper]

Presently, on agent reboot, the agent will come up with a new ID. This presents a problem because it means that Mesos will never be able to report that a LOST task as gone, and the instance will forever be in the Unreachable state, even though it's associated reservation is reoffered. We expect this to be fixed by https://issues.apache.org/jira/browse/MESOS-6223, which is slated to be released with Mesos 1.3.0.

Until then, we plan to monitor the offer stream, watch for reservations, and then check associated instance state. If the instance state is unreachable, then we will emit a terminal TASK_GONE mesos update so that the task can transition appropriately.

[timcharper]

https://phabricator.mesosphere.com/D566 submitted; I've confirmed it to work manually in a local cluster setup.

[meichstedt]

Note: This issue has been migrated to https://jira.mesosphere.com/browse/MARATHON-2311. For more information see https://groups.google.com/forum/#!topic/marathon-framework/khtvf-ifnp8.