Use efficient PubSub on Postgres

[mgax]

Postgres implements PubSub channels with the NOTIFY and LISTEN statements. This means that a worker doesn't have to periodically poll the database to get more tasks, it can just wait for notifications.

This might actually be counter-productive for high-volume setups – if there are always tasks in the queue, polling will always return a task, and the notifications are just overhead – but for low-volume sites, where a single worker spends most of its time idle, this would save a bunch of CPU cycles.

[RealOrangeOne]

It'd be nice to avoid the polling, I agree.

The downside is we'd need to bypass the ORM, as it's platform-specific (and not in django.contrib.postgres). At a certain scale though, it may be necessary.

I'm not sure how NOTIFY and LISTEN deal with transactions - that'll be something to test.

[hooverdc]

Hmm, is this conceptually different enough to be its own backend?

[RealOrangeOne]

It depends. Given the majority of the functionality would be shared, it ought to be ok to be a configuration change. Whether it's opt-in or opt-out will depend on what impact it has, especially around migrations and locking.