Investigate the need in creating the task management infra around celery

[bkyryliuk]

Problem statement:
There are the tools / services that require to execute the tasks / jobs remotely and report back their status and results. There is a need to have some execution guarantees, monitoring, logging and ability to manage the state of the queue in order to effectively maintain the tools.

Issues with Celery:

• no visibility into the tasks / progress
• no logging
• no execution guarantees
• no viable API to kill the task

Possible solutions

Separate service:

• build a service that will execute the shell commands
• with a simple api: schedule task/s, get list of tasks (with filtering), change task/s status, get task/s progress, delete task/s
• with logging, monitoring in place
• open source to support the airflow and caravel communities

Orchestrate the celery for our needs

• update celery to put additional data into the queue:
• keep the running tasks in the queue
• add heartbeat functionality
• add logging
• build a tool to introspect and monitor the celery queue
• build a tool to manage the tasks

[bkyryliuk]

Not needed in the short term, closing.