Workers keep dying D:

[ghost]

This has been a long standing issue for a while. Occasionally the workers responsible for processing tracks will suddenly fall over. There has been an attempt at alerting users that there's a problem by returning a 418 error when a track file is not found. There needs to be some sort of detection system to detect when these 418's are fired a few times and restart the workers

[EventideGlow]

@logic-dev Have you found anything interesting in the error logs when the workers crash? It would be nice to figure out what causes this, in addition to setting up more reliable monitoring.

The workers are currently monitored with Supervisord - sounds like our config for that in Ansible might need tweaking.

[EventideGlow]

This might be related to #114 - if there are issues with what the workers are doing, normally, those would be getting logged in the failed_jobs table.