Customizable Apache Mesos task executor. It allows controlled graceful task shutdown and performing various additional actions during the task lifecycle, by providing hook mechanisms (see hook package).
Executor uses Mesos HTTP API to communicate with agent. When executor is started, before doing anything else, it tries to subscribe to Mesos agent. After successful subscription it waits for events from agent to handle. During the whole process executor keeps connection with a Mesos agent. When connection is lost it tries to reconnect for a configured time and when it fails to do so it tries to finish the started task and stops the whole executor process.
Task is started when Event_LAUNCH is received with required TaskInfo. Before
starting the received command executor fires BeforeTaskStartEvent event hook,
and if any of registered hooks fail to do their jobs, it stops the execution process
and fails. This hook can be used to modify the environment of the task by returning
formatted variable strings ("VAR=value"). Right after starting the command,
executor fires AfterTaskStartEvent event hook - and again if any of the hooks fail,
executor fails also. It is worth noting that executor may exit without even
starting a task.
Executor may exit in the following cases:
- started tasks fail to start or run - executor quits with
TASK_FAILEDsent to Mesos agent - started tasks exit with 0 return code - executor quits with
TASK_FINISHEDsent to Mesos agent - executor receives
Event_SHUTDOWNorEvent_KILL- executor quits withTASK_KILLEDsent to Mesos agent
Executor always fires BeforeTerminateEvent event hook when exiting - regardless
of whether it started a task or not.
Graceful Shutdown is a feature to minimize task killing impact on other systems. It is performed in the following steps:
- Call all hooks with
BeforeTerminateEvent. - Sent SIGTERM to process tree.
- Wait
KillPolicyGracePeriod(can be overridden with Task Kill Policy Grace Period). - Sent SIGKILL to process tree.
Executor can be configured to exclude certain processes from SIGTERM signal. Provide
process names to exclude in ALLEGRO_EXECUTOR_SIGTERM_EXCLUDE_PROCESSES environment variable
as a comma-separated string. This feature requires pgrep -g to be available on the machine.
By default executor forwards service stdout/stderr to its own standard streams. It can however redirect them to data processing pipeline - Logstash. This requires you to set up the connection to the Logstash service in the executor's environmental variables:
ALLEGRO_EXECUTOR_SERVICELOG_LOGSTASH_PROTOCOL="tcp" # tcp or udp
ALLEGRO_EXECUTOR_SERVICELOG_LOGSTASH_ADDRESS="localhost:1234" # host and portCurrently, the executor is able to parse and send only logs in the logfmt
format. To enable log scraping you need to set log-scraping label in Mesos
TaskInfo to logfmt. For more information see documentation of servicelog
package.
Executor supports integration with external system via hooks. The hook is an interface
with functions that will be called when specific actions occur. To use hooks just
implement hook.Hook and plug it into hook.Manager.
Hooks calls are blocking.
Integration with Consul is based on a hook. It mimics the behavior of
allegro/marathon-consul.
Task is registered in Consul once it becomes healthy and deregistered before kill.
Required task metadata such as name, labels and ports are obtained from task definition.
Service name is taken from consul label.
Labels are transformed to Consul tags only when value is equal tag. Client does not use any ACL Token by default,
this can be changed by setting CONSUL_TOKEN environment variable.
VaaS integration is based on a hook.
Task is registered once it becomes healthy and deregistered before kill.
Task’s first port will be registered under director provided in a label named director.
If task has defined weight in a label it will be used. Weight could be overridden
with VAAS_INITIAL_WEIGHT environment variable.
If task is a canary instance (has non empty canary label) backend is marked
as a canary.
To run executor tests locally you need following tools installed:
If you want to test executor locally, you will need additionally:
Executor offers a debug mode that provide extended logging and capabilities during
runtime. Enabling this can significantly increase the amount of resources the
executor needs to operate, so do not turn this on, when it is not needed. To enable
debug mode add -debug flag to executor command or set ALLEGRO_EXECUTOR_DEBUG
environment variable to true.
To create your Vagrant environment execute following command in project root folder:
$ vagrant up
It will create a virtual machine with Apache Mesos and Marathon installed and running on it. Mesos UI will be available on http://localhost:5050 and Marathon UI on http://localhost:8080.
If you want to test executor on Vagrant Mesos you will have to create release build of it. To do this, execute the following command:
$ make release
Binary will be immediately available on virtual machine on following address:
You can use above address to configure your Marathon application to be executed by your freshly build executor. An example application is available in marathon-test-app.json
Executor configuration can be altered via environment variables in the following way:
ALLEGRO_EXECUTOR_STATE_UPDATE_BUFFER_SIZE="2048"
ALLEGRO_EXECUTOR_STATE_UPDATE_WAIT_TIMEOUT="3s"
sets the StateUpdateBufferSize Config property to 2048, and StateUpdateWaitTimeout
to 3 seconds. For all available settings and their defaults see
executor.go.
Additionally, a Consul instance is available for testing purposes, its logs can be viewed by running:
$ vagrant ssh
vagrant@localhost:~$ sudo supervisorctl tail -f consul
- Executor may not send a SIGKILL to process tree after grace period, so service process may be still running when executor finishes. To clean up executor and launched tasks properly use pid isolator.
See CONTRIBUTING for more details and code of conduct.
Mesos Executor is distributed under the Apache 2.0 License.