Mutate API

It's inevitable: you will have cases where you enqueue bad data or have jobs which need pruning, migration or some other rare use case. For this, Faktory provides low-level data management APIs so you can script certain repairs or migrations.

Please be warned: MUTATE commands can be slow and/or resource intensive. They should not be used as part of your application logic.

MUTATE

The MUTATE command takes a single JSON hash with three parts:

1. A target structure, legal values are "retries", "scheduled", "dead"
2. A command to perform on matching jobs: "kill", "discard" or "requeue"
3. A filter to match jobs within that structure by JIDs, regexp or exact jobtype

Structures

Only the Retries, Dead and Scheduled Sets may be targeted. Queues cannot be mutated, you can use the Web UI to clear them or remove entries manually; this functionality is not exposed to MUTATE today.

Why? Queues are meant to be cleared quickly. Only the three Sets should be holding jobs for any length of time so there's no reason to target queues for migration or repair.

Commands

• kill - moves the job from the target structure to the Dead set. Faktory will not touch it further, you can manually process it via the Web UI some point in the future.
• discard - throw the job away, POOF it's gone.
• requeue - immediately move the job back into its queue for processing by a worker

Filter

You can filter jobs based on three criteria:

1. Jobtype - must be an exact jobtype, not a pattern or substring. Fast.
2. An array of JIDs - passing one JID is fast, passing more than one becomes much slower.
3. Regexp - this is a low-level match against the entire job payload, tricky but fast.

If you want to use an array of JIDs, I'd recommend also supplying a Jobtype to prune possible matches first.

Examples

Find all scheduled jobs with type QuickbooksSyncJob and discard them:

MUTATE {"cmd":"discard","target":"scheduled","filter":{"jobtype":"QuickbooksSyncJob"}}

Clear the Retry queue completely:

MUTATE {"cmd":"discard","target":"retries","filter":{"regexp":"*"}}
MUTATE {"cmd":"discard","target":"retries"}

Send a two specific JIDs in the Retry queue to the Dead queue, note the use of jobtype to narrow the match candidates:

MUTATE {"cmd":"kill","target":"retries","filter":{"jobtype":"QuickbooksSyncJob", "jids":["123456789", "abcdefgh"]}}

Enqueue all retries with a first argument matching "bob":

MUTATE {"cmd":"requeue","target":"retries","filter":{"regexp":"*\"args\":[\"bob\"*"}}

Note the regexp filter scans the entire job payload and can be tricky to get right, for instance you'll probably need * on both sides. The regexp filter option is passed to Redis's SCAN command directly, read the SCAN documentation for further details.

Client Implementation

Please see Faktory's Go client package or Ruby worker package for examples of how libraries can implement MUTATE.

Queues

Faktory queues are different from mutable sets and so have a different set of operations which you can perform.

Pausing

Queues may be paused (no job can be fetched from them while paused) or unpaused (resume fetching). * means all queues but does not act as a wildcard, "foo*" will not match "foobar".

queue pause bulk another_queue
queue pause *
queue unpause *
queue unpause bulk

See https://github.com/contribsys/faktory/blob/62ec7321e34a6bd75a1c8c5742ee1dbaab0e9fba/client/client.go#L395.

Remove

Queues can be removed which deletes all jobs within them. It does not stop currently executing jobs from those queues.

queue remove bulk
queue remove *

Notes

• MUTATE is best effort; it makes no attempt to lock or make its actions atomic. Since job data is constantly changing, race conditions will be possible and even common.