i852 Improve slow video processing speeds

[bkiahstroud]

Story

Refs

• 🐛 Video processing speeds #852

@orangewolf and I determined that the slowness identified in #852 is due to insufficient CPU resources; ffmpeg commands run on large files were being throttled and thus running extremely slowly.

To solve this, we decided that when an audio or video derivative job is triggered, we should put in into a separate Sidekiq queue (i.e. :resource_intensive). We'll have the default worker(s) run all queues other than :resource_intensive. We'll also have a new, separate worker that runs all the other queues plus the :resource_intensive queue. The new worker will have 4x the CPU resources that the default workers have, but only 1 thread.

This effectively creates a powerful "slow lane". :resource_intensive jobs will slow down the separate worker while they're running, but they won't bog down all the jobs since the other workers are still running.

Part 1

This PR implements the first part of the solution described above. Specifically, it:

• Adds the new :resource_intensive Sidekiq queue
• Builds out the logic that diverts only audio and video files on the :resource_intensive queue

Expected Behavior Before Changes

Video and audio files get processed by the CreateDerivativesJob in the :default Sidekiq queue

Expected Behavior After Changes

Video and audio files get processed by the CreateLargeDerivativesJob in the :resource_intensive Sidekiq queue

Screenshots / Video

Notes

Flow of logic introduced in this PR

flowchart TD
    0([CreateDerivativesJob#perform_later]) -->
    S[CreateDerivativesJob#perform]
    V{Is audio or video?}
    D[CreateDerivativesJobDecorator]
    L{Is CreateLargeDerivativesJob?}
    H([Hyrax logic])
    J[Spawn CreateLargeDerivativesJob]
    S --> D
    D --> L
    L --> |No|V
    L --> |Yes|H
    V --> |No|H
    V --> |Yes|J
    J --> |#super|S
    style H stroke:#f66,stroke-width:2px
    style V stroke:yellow
    style L stroke:yellow
    style 0 stroke:#0f0,stroke-width:2px

Loading