Skip to content

Commit

Permalink
feat(doc): enable FTS substring search (#166)
Browse files Browse the repository at this point in the history
  • Loading branch information
DavidePrincipi authored Feb 6, 2025
1 parent 85f8bd3 commit 749d57d
Show file tree
Hide file tree
Showing 2 changed files with 51 additions and 6 deletions.
45 changes: 41 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,7 @@ by `myselector`,
3. add a DNS TXT record to `mydomain.example.com`, as described in
`/var/lib/rspamd/dkim/myselector.txt`

## Configuration override for ClamAV unofficial signatures
## Configuration override for ClamAV unofficial signatures

Changes to `clamav-unofficial-sigs` configuration are volatile. When the
`clamav` container is stopped, local configuration changes are lost.
Expand Down Expand Up @@ -330,10 +330,47 @@ Migration notes:
`rspamd/RecipientWhiteList`.

1. New Full Text Search (FTS) engine Flatcurve. Old FTS indexes are
excluded from the migration. To massively rebuild the search indexes
run the following command during system idle time:
excluded from the migration. Refer to **Full Text Searche (FTS)**
section below.

podman exec dovecot nice doveadm index -A -q '*'
## Full Text Search (FTS)

The default configuration enables full-text search over mail headers and
body. To keep the FTS index size small, only exact-word tokens are indexed
by default.

To enable complete substring searches, the indexes must be rebuilt, which
significantly increases disk space usage.

### Enable FTS substring search

It is advisable to run this procedure outside of working hours because
mailbox reindexing has high system resource demands.

Edit the module's `state/environment` file by adding the following line:

DOVECOT_FLATCURVE_SUBSTRING_SEARCH=yes

Reload the Dovecot container:

systemctl --user reload dovecot

Open a shell in the Dovecot container:

podman exec -ti dovecot ash -l

Calculate the size of the current FTS indexes (in MB):

find . -type d -name fts-flatcurve -print0 | xargs -r -0 -- du -m | awk '{ tot += $1 } ; END { print tot }'

Remove existing `fts-flatcurve` directories:

find . -type d -name fts-flatcurve -exec rm -rf '{}' \;

To rebuild the search indexes in bulk, run the following command during
system idle time:

nice doveadm index -A -q '*'

## UI translation

Expand Down
12 changes: 10 additions & 2 deletions dovecot/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,8 +61,16 @@ Private TCP ports:
documentation](https://doc.dovecot.org/settings/core/#core_setting-default_process_limit).
The `default_client_limit` parameter is proportionally set to 10 times this value.
- `DOVECOT_SHAREDSEEN`, default empty. If set, the SEEN flag of shared folders is stored in a `dovecot.pvt*` index file, for every user
- `DOVECOT_FLATCURVE_SUBSTRING_SEARCH`, default `no` (`yes` to enable). If enabled, allows substring searches (RFC 3501 compliant). However, this requires significant additional storage space. Many users today expect "Google-like" behavior, which is prefix searching, so substring searching is arguably not the modern expected behavior anyway. Therefore, even though it is not strictly RFC compliant, prefix (non-substring) searching is enabled by default.

- `DOVECOT_FLATCURVE_SUBSTRING_SEARCH`, default `no` (`yes` to enable).
Many users today expect "Google-like" behavior, which is prefix
searching, so substring searching is arguably not the modern expected
behavior anyway. Therefore, even though it is not strictly RFC
compliant, prefix (non-substring) searching is enabled by default.

If enabled, allows substring searches (RFC 3501 compliant). However,
this requires rebuilding the existing fts-flatcurve indexes with
significant additional storage space.

## Logs

The container sends log messages to `syslog`. Increase log verbosity by
Expand Down

0 comments on commit 749d57d

Please sign in to comment.