[enhancement]: Rate limiting sends

[nomadturk]

Which feature or improvement would you like to request?

I would like to impose rate limiting when a new email is being sent, on a per-domain, per-mailbox level.
(Forwards or DSNs should not count)

We should be able to either reject or re-queue when they hit rate limits.
This would help prevent email bursts and virus related outbreaks as well as protecting the domain/IPs.

Is your feature request related to a problem?

I'm having a problem with...

Code of Conduct

• I agree to follow this project's Code of Conduct

[nomadturk]

Regarding this,

Just today, one of the client had some of their passwords leaked, on a different server.
As a result, the spammer tried to send a few hundred thousand emails.

Thankfully, since I had hourly limits on a per-mailbox /per-domain / per-customer basis, they were only able to send ~100 emails before the system stopped the outbreak and we intervened.

This is an important feature to have.

I know there is a "Rate Limit" feature in the interface. But it's unrelated.
I don't want to limit imap checks though. I want to rate limit email sends. And this limit should be adjustable on an account/domain and tenant basis.

[osebastian]

+1.

This, along with the ability to pause on a per-account/domain basis in the event of abuse, is a critical feature that is absolutely essential.

Password/credential leakage and the abuse of these by spammers are very common scenarios in the wild.

[mdecimus]

This is already possible, check the documentation for outbound rate limits.

For example, to throttle by sender address:

[[queue.throttle]]
key = ["sender"]
rate = "25/1h"
enable = true

Or by sender domain:

[[queue.throttle]]
key = ["sender_domain"]
rate = "25/1h"
enable = true

Or a combination of sender and recipient domain:

[[queue.throttle]]
key = ["sender", "rcpt_domain"]
rate = "25/1h"
enable = true

[nomadturk]

Yeah...

You can't do this via Sieve. It can be OK if you are running 5-10 domains and 20-30 users.
But anything after that, this only becomes an unmanageable nightmare.

This should be an API endpoint and part of webadmin for UI users.
With different rate limiting per user. (and tenant or domain)

[mdecimus]

You can't do this via Sieve.

Do you mean enforcing rate limits from Sieve?

It can be OK if you are running 5-10 domains and 20-30 users.

The queue throttle setting is a global setting and does not need to be configured per user. For example, the sender example above limits any account to send at maximum 25 messages per hour. The sender + rcpt_domain example limits any account to send 25 messages per hour to the same domain.
You can then use expressions if you want to selectively apply these throttles to certain accounts.

[nomadturk]

That... is not practical.

In many scenarios, it does need to be configured per user.
You can't have a global rule for all users.

On a busy server, how does one implement such rules to hundreds of domains, thousand of users?
And try to update that. On each node of the cluster?

There are times, even if we let a single user send X amount of emails, we can limit the number of emails sent from the domain or even from the tenant.

Here is an example from Plesk
There are overridable limits on a per-mailbox/per-domain/per-tenant basis.

Also...

[[queue.throttle]]
key = ["sender_domain"]
rate = "25/1h"
enable = true

• Let's say you have a service account sending emails X emails per hour.
  Why should it have the same limits as the other users under the same domain?

These rules can not be added as sieve filters.
They should be DB entries, populated to Redis or something like that.

And we should be able to call these changes via an API endpoint.

[osebastian]

@mdecimus, thank you for your quick reply. Since I posted, I've already found inbound/outbound throttling and started using them. It's a great feature, and you can make some granular rules per domain, sender, ip etc.

However, I still have a few questions, and I'd appreciate it greatly if you could clarify them.

1. About pausing:
   Currently, we change passwords if we notice abuse from an acocunt. Is there any other way to do this/pause an account?

2. About monitoring:
   Can we check anywhere how much of that quota/throttling has been used?

3. About Integration of both:
   In the scenario where someone's smtp/account credentials are leaked or abused, it would be great to somehow be able to integrate monitoring and intervention. I already checked documentation for Telemetry (metrics+tracing).

The best case scenario for us would be monitoring everything in Grafana/GrafanaCloud (usage/mails sent per domain, address etc) and when abuse is noticed - automatically pause account via API/Webhook/something else.

Is this possible/is there a future roadmap for this/can I post an enhancement request?

Thank you for your patience!

[osebastian]

That... is not practical.

In many scenarios, it does need to be configured per user. You can't have a global rule for all users.

On a busy server, how does one implement such rules to hundreds of domains, thousand of users? And try to update that. On each node of the cluster?

There are times, even if we let a single user send X amount of emails, we can limit the number of emails sent from the domain or even from the tenant.

Here is an example from Plesk There are overridable limits on a per-mailbox/per-domain/per-tenant basis.

Also...

[[queue.throttle]]
key = ["sender_domain"]
rate = "25/1h"
enable = true

• Let's say you have a service account sending emails X emails per hour.
  Why should it have the same limits as the other users under the same domain?

These rules can not be added as sieve filters. They should be DB entries, populated to Redis or something like that.

And we should be able to call these changes via an API endpoint.

@nomadturk I think you can rate limit on a more granular level using The queue.throttle[].key as following:

remote_ip: The remote IP address.
local_ip: The local IP address (only available when a source IP is configured).
mx: The remote host's MX hostname.
sender: The return path specified in the MAIL FROM command.
sender_domain: The domain component of the return path specified in the MAIL FROM command.
rcpt: The recipient's address specified in the RCPT TO command.
rcpt_domain: The domain component of the recipient's address specified in the RCPT TO command.

So you can limit per sender(mailbox)/domain/combination/et cetera.

Of course, I personally think it would be better from a UX perspective to somehow integrate this in the management under domain/acocunts settings. And also create some form of "templates" for new accounts/domains with predefined rate limits etc.

[nomadturk]

@osebastian

Yes, I know about the /settings/smtp-out-throttle page.
I've rate limits for outbound.

But it's neither sufficient, nor gives me any visibility.

We should also be able to create exclusions without messing with that page.
This should be a setting under domains and mailboxes. Not a global admin setting.

If you have tenants, how are you going to let them change that setting for themselves?
Also, it doesn't make much sense if you can't configure that setting on a more granular level.

And in the event of a user hitting these throttles X times, it should fire a webhook, alert etc. for human intervention.

[osebastian]

@nomadturk I agree. I think from a UX point of view, the best would be to:

1. Manage limits globally somewhere by server admin. The current throttling under inbound/outbound settings should suffice.
2. Manage limits on a per domain/account/tenant basis by server admin. This could be integrated in the same way disk quota is set when creating new accounts, under the "limits tab"
3. Manage limits by tenants/accounts themselves, if they want to set a limit smaller than the one set by server admin

This + visibilitiy. There should be a way to monitor disk quota + sending limits etc (and how much is utilized/left). Monitoring + automatic intervention (via grafana+api or something else) is absolutely necessary if using stalwart with more than a few accounts/domains that you can personally manage.

[nomadturk]

@osebastian

Webadmin should have these too. Right.
But I'm mostly after the API approach :)

A domain admin or a tenant should be able to change the limits, with the upper limit being the limit we set it up for their account. The global limits should be over-ridable. There shouldn't be a one limit to rule them all.

And yes, when this is something other than a personal server... Quota, limits, alerts, webhooks these are all essential to operate an email server.

[nomadturk]

Unfortunately the outbound rate limit did not work as one would expect.
Somehow it just crippled the server as well as database.

Under SMTP > Outbound Throttles
I had 2 rate limits, 1 hourly and 1 daily limit.
They were supposed to be rate limits, per sender, for outgoing emails

These users have not sent that many emails at all.

But, since the internal delivery is also counted as outbound, when busy hour started, internal delivery just killed the whole server.
Why does Stalwart even check from = <>?

2024-12-05T04:35:35Z INFO Rate limit exceeded (queue.rate-limit-exceeded) queueId = 212008089975525379, from = <>, to = ["user@domain.com"], size = 3168, total = 1, id = "throttle-outbound-sender-daily", limit = [1600, 86400000ms]
2024-12-05T04:35:35Z WARN Rate limit exceeded (delivery.rate-limit-exceeded) queueId = 212008089975525379, from = <>, to = ["user@domain.com"], size = 3168, total = 1, id = "throttle-outbound-sender-daily", nextRetry = 2024-12-06T00:00:00Z

or

2024-12-05T12:10:52Z INFO Rate limit exceeded (queue.rate-limit-exceeded) queueId = 212019450451341907, from = "user2@domain2com", to = ["somerecipient@somedomain.com"], size = 1784, total = 1, id = "throttle-outbound-sender-daily", limit = [1600, 86400000ms]
2024-12-05T12:10:52Z INFO Rate limit exceeded (queue.rate-limit-exceeded) queueId = 212065378010928303, from = <>, to = ["user2@domain2.com"], size = 3110, total = 1, id = "throttle-outbound-sender-daily", limit = [1600, 86400000ms]

The server

and the DB

...