Package version stuck on 1.5.5-1 on debian 12 #178

seb1k · 2024-12-01T10:28:36Z

Hi,

Thanks for this amazing tool !

It really improve search on my roundcube server :

But on debian 12 (most common linux distro) the last version is stuck on 1.5.5 (april 2022)
That's anoying, this prevents an easy installation ( apt-get install dovecot-fts-xapian )

Package info here : https://packages.debian.org/bookworm/dovecot-fts-xapian

Are the developer are aware of this issue ?

Thanks a lot !

grosjo · 2024-12-01T11:56:58Z

Latest debian include latest plugin
https://packages.debian.org/unstable/mail/dovecot-fts-xapian

@jnahmias may comment on this request to update older releases

jnahmias · 2024-12-09T01:45:52Z

But on debian 12 (most common linux distro) the last version is stuck on 1.5.5 (april 2022) That's anoying, this prevents an easy installation ( apt-get install dovecot-fts-xapian )

I've uploaded 1.8 to bookworm-backports. See https://backports.debian.org/Instructions/ for details on how to use it on your Debian 12 machine.

adixp2010 · 2024-12-09T15:16:12Z

I tried like in Instruction, but now have problem with dovecot-abi-2.3.abiv19, it's looks like it doesn't support 2.3.ABIv21 which is installed with dovecot 2.3.21 (default version installed in debian 12).

`# apt install dovecot-fts-xapian/bookworm-backports
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Selected version '1.8-1~bpo12+1' (Debian Backports:stable-backports [amd64]) for 'dovecot-fts-xapian'
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 dovecot-fts-xapian : Depends: dovecot-abi-2.3.abiv19
E: Unable to correct problems, you have held broken packages.
# dovecot --version
2.3.21.1 (d492236fa0)`

seb1k · 2024-12-09T17:08:27Z

Hi !
Thanks for the upload.

The installation was fine for me, usage too.

When I recreated the index DB to see changes, these problems appear :

1 ) "Too many open files" errors :

doveadm(seb): Error: net_connect_unix(/run/dovecot/decode2text) failed: Too many open files
doveadm(seb): Error: socket(/run/dovecot/decode2text) failed: Too many open files

Resolved with " ulimit -n 4096" + change in dove service config

"+-10 randoms buggy emails"

doveadm(seb): Error: Mailbox INBOX: open(/home/seb/Maildir/cur/1727626041.M340684P111729.Sebserv,S=36112,W=37029:2,S) failed: Too many open files
doveadm(seb): Error: Mailbox INBOX: UID=8465: Failed to read stream: Mailbox INBOX: open(/home/seb/Maildir/cur/1727626041.M340684P111729.Sebserv,S=36112,W=37029:2,S) failed: Too many open files

Solved by deleting the "buggy" emails.

"Xapian::DatabaseOpeningError"

terminate called after throwing an instance of 'Xapian::DatabaseOpeningError'
terminate called recursively
Aborted

I retry multiple time the indexing until the end... (I could see the index file growing each time)

And it finished normally ( after 7-8 indexing )

For info : xapian-tools (1.4.22-1 Debian:12.8/stable [amd64])

seb1k · 2024-12-09T23:34:03Z

For debugging, here is my syslog : (it seems to be a memory error)

...
2024-12-09T11:32:05.920969+01:00 Sebserv xapian-docswriter: DW #3 (Sent,/home/seb/Maildir/xapian-indexes/db_a2e9f60aeca81362d10f0000877e8673) - Committing 1 docs due to low free memory (203 MB vs 500 MB) or Cached docs > 4000
2024-12-09T11:32:06.103380+01:00 Sebserv xapian-docswriter: DW #3 (Sent,/home/seb/Maildir/xapian-indexes/db_a2e9f60aeca81362d10f0000877e8673) - Committing 1 docs due to low free memory (197 MB vs 500 MB) or Cached docs > 4000
2024-12-09T11:32:06.229955+01:00 Sebserv xapian-docswriter: DW #1 (Sent,/home/seb/Maildir/xapian-indexes/db_a2e9f60aeca81362d10f0000877e8673) - Committing 1 docs due to low free memory (197 MB vs 500 MB) or Cached docs > 4000
2024-12-09T11:32:06.382388+01:00 Sebserv xapian-docswriter: DW #3 (Sent,/home/seb/Maildir/xapian-indexes/db_a2e9f60aeca81362d10f0000877e8673) - Committing 1 docs due to low free memory (196 MB vs 500 MB) or Cached docs > 4000
2024-12-09T11:32:06.896076+01:00 Sebserv xapian-docswriter: DW #1 (Sent,/home/seb/Maildir/xapian-indexes/db_a2e9f60aeca81362d10f0000877e8673) - Committing 1 docs due to low free memory (191 MB vs 500 MB) or Cached docs > 4000
2024-12-09T11:32:06.939207+01:00 Sebserv xapian-docswriter: DW #2 (Sent,/home/seb/Maildir/xapian-indexes/db_a2e9f60aeca81362d10f0000877e8673) - Committing 1 docs due to low free memory (189 MB vs 500 MB) or Cached docs > 4000
2024-12-09T11:32:07.727345+01:00 Sebserv xapian-docswriter: DW #2 (INBOX.Autres,/home/seb/Maildir/xapian-indexes/db_6847f22aa1cee362d3020000274c5d4b) - Committing 1 docs due to low free memory (169 MB vs 500 MB) or Cached docs > 4000
2024-12-09T11:32:07.779071+01:00 Sebserv xapian-docswriter: DW #2 (INBOX.Autres,/home/seb/Maildir/xapian-indexes/db_6847f22aa1cee362d3020000274c5d4b) - Committing 1 docs due to low free memory (169 MB vs 500 MB) or Cached docs > 4000
2024-12-09T11:32:07.950093+01:00 Sebserv xapian-docswriter: DW #3 (INBOX.Autres,/home/seb/Maildir/xapian-indexes/db_6847f22aa1cee362d3020000274c5d4b) - Committing 1 docs due to low free memory (163 MB vs 500 MB) or Cached docs > 4000
2024-12-09T11:32:08.005981+01:00 Sebserv xapian-docswriter: DW #3 (INBOX.Autres,/home/seb/Maildir/xapian-indexes/db_6847f22aa1cee362d3020000274c5d4b) - Committing 1 docs due to low free memory (163 MB vs 500 MB) or Cached docs > 4000
2024-12-09T11:32:08.059916+01:00 Sebserv xapian-docswriter: DW #3 (INBOX.Autres,/home/seb/Maildir/xapian-indexes/db_6847f22aa1cee362d3020000274c5d4b) - Committing 1 docs due to low free memory (163 MB vs 500 MB) or Cached docs > 4000
2024-12-09T11:32:08.191441+01:00 Sebserv xapian-docswriter: DW #1 (INBOX.Autres,/home/seb/Maildir/xapian-indexes/db_6847f22aa1cee362d3020000274c5d4b) - Committing 1 docs due to low free memory (163 MB vs 500 MB) or Cached docs > 4000
2024-12-09T11:32:08.977906+01:00 Sebserv xapian-docswriter: DW #3 (INBOX,/home/seb/Maildir/xapian-indexes/db_1197e408e8a81362cc0f0000877e8673) - Can't lock the DB : DatabaseLockError - Unable to get write lock on /home/seb/Maildir/xapian-indexes/db_1197e408e8a81362cc0f0000877e8673: already locked
2024-12-09T11:32:08.978571+01:00 Sebserv xapian-docswriter: DW #3 (INBOX,/home/seb/Maildir/xapian-indexes/db_1197e408e8a81362cc0f0000877e8673) - Can't lock the DB : DatabaseLockError - Unable to get write lock on /home/seb/Maildir/xapian-indexes/db_1197e408e8a81362cc0f0000877e8673: already locked
2024-12-09T11:32:08.979082+01:00 Sebserv xapian-docswriter: DW #3 (INBOX,/home/seb/Maildir/xapian-indexes/db_1197e408e8a81362cc0f0000877e8673) - Can't lock the DB : DatabaseLockError - Unable to get write lock on /home/seb/Maildir/xapian-indexes/db_1197e408e8a81362cc0f0000877e8673: already locked
2024-12-09T11:32:08.982405+01:00 Sebserv xapian-docswriter: DW #3 (INBOX,/home/seb/Maildir/xapian-indexes/db_1197e408e8a81362cc0f0000877e8673) - Can't lock the DB : DatabaseLockError - Unable to get write lock on /home/seb/Maildir/xapian-indexes/db_1197e408e8a81362cc0f0000877e8673: already locked
2024-12-09T11:32:08.982772+01:00 Sebserv xapian-docswriter: DW #3 (INBOX,/home/seb/Maildir/xapian-indexes/db_1197e408e8a81362cc0f0000877e8673) - Can't lock the DB : DatabaseLockError - Unable to get write lock on /home/seb/Maildir/xapian-indexes/db_1197e408e8a81362cc0f0000877e8673: already locked
2024-12-09T11:32:08.982976+01:00 Sebserv xapian-docswriter: DW #3 (INBOX,/home/seb/Maildir/xapian-indexes/db_1197e408e8a81362cc0f0000877e8673) - Can't lock the DB : DatabaseLockError - Unable to get write lock on /home/seb/Maildir/xapian-indexes/db_1197e408e8a81362cc0f0000877e8673: already locked
2024-12-09T11:32:08.983727+01:00 Sebserv xapian-docswriter: DW #3 (INBOX,/home/seb/Maildir/xapian-indexes/db_1197e408e8a81362cc0f0000877e8673) - Can't lock the DB : DatabaseLockError - Unable to get write lock on /home/seb/Maildir/xapian-indexes/db_1197e408e8a81362cc0f0000877e8673: already locked
2024
...

grosjo · 2024-12-10T16:11:23Z

It seems you run on a very low memory machine ?

seb1k · 2024-12-10T16:19:53Z

more or less ... (2Gb memory container, 90% free), this has never been a problem before.

grosjo · 2024-12-10T16:38:21Z

Have you set vsz_limit in your dovecot.conf (to 1G or so) ?

seb1k · 2024-12-10T16:59:54Z

service indexer-worker {
    # Increase vsz_limit to 2GB or above.
    # Or 0 if you have rather large memory usable on your server, which is preferred for performance)
    vsz_limit = 2G
    process_limit = 0
}

grosjo · 2024-12-11T04:45:24Z

Please retry with latest git

seb1k · 2024-12-11T09:32:17Z

Recompiled with last git :
Still lot of errors :

Error: socket(/run/dovecot/decode2text) failed: Too many open files
doveadm(seb): Error: net_connect_unix(/run/dovecot/decode2text) failed: Too many open files

Solved with "ulimit -n 4096"

The indexing keep crashing after a while :

root@Sebserv:~/fts-xapian# doveadm index -u seb \*
terminate called after throwing an instance of 'Xapian::DatabaseOpeningError'
terminate called recursively
Aborted

grosjo · 2024-12-11T10:14:10Z

Is the problem coming from decode2text of fts-xapian ?

seb1k · 2024-12-11T11:02:37Z

root@Sebserv:/etc/apt# apt remove dovecot-fts-xapian
...
Removing dovecot-fts-xapian (1.8-1~bpo12+1) ...

root@Sebserv:/etc/apt# apt install dovecot-fts-xapian
...
Unpacking dovecot-fts-xapian (1.5.5-1+b2) ...

Index flushing
doveadm fts rescan -u seb

Index creation
doveadm index -u seb \*

Finished properly ... So it seems fts-xapian

grosjo · 2024-12-29T02:00:49Z

https://tracker.debian.org/news/1592540/accepted-dovecot-fts-xapian-18-1bpo121-source-amd64-into-stable-backports/

grosjo · 2024-12-29T02:01:22Z

@jnahmias Can you upgrade to 1.8.1 ? Thank you so much

grosjo · 2025-01-02T00:22:37Z

1.8.3 actually

seb1k · 2025-01-07T18:05:46Z

I compiled the last git.
No bug, way faster and indexes are small compared to previous version.
Can't wait to see an updated version available on official repo.

Merci beaucoup, tres beau debugage !

grosjo · 2025-01-09T00:17:13Z

@jnahmias 1.8.4 version released - Thank you so much

grosjo · 2025-01-14T14:44:21Z

1.8.5 released

reox · 2025-01-15T08:04:59Z

btw; what I do on my Debian Bookworm server, is to build a fresh version with this script: https://gist.github.com/reox/efa3686c3ac9b95d1fac8b3e768bef9a
If you are an expert on gbp (I'm not...) you may be able to condense it even more.

grosjo · 2025-02-02T03:35:57Z

@jnahmias 1.9 released

grosjo · 2025-02-22T04:11:46Z

@jnahmias 1.9.1 released to include compatibility with dovecot 2.4

grosjo added a commit that referenced this issue Dec 11, 2024

Issue #178

27e02c2

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Package version stuck on 1.5.5-1 on debian 12 #178

Package version stuck on 1.5.5-1 on debian 12 #178

seb1k commented Dec 1, 2024

grosjo commented Dec 1, 2024

jnahmias commented Dec 9, 2024

adixp2010 commented Dec 9, 2024 •

edited

Loading

seb1k commented Dec 9, 2024 •

edited

Loading

seb1k commented Dec 9, 2024 •

edited

Loading

grosjo commented Dec 10, 2024

seb1k commented Dec 10, 2024

grosjo commented Dec 10, 2024

seb1k commented Dec 10, 2024 •

edited

Loading

grosjo commented Dec 11, 2024

seb1k commented Dec 11, 2024

grosjo commented Dec 11, 2024

seb1k commented Dec 11, 2024

grosjo commented Dec 29, 2024

grosjo commented Dec 29, 2024

grosjo commented Jan 2, 2025

seb1k commented Jan 7, 2025

grosjo commented Jan 9, 2025

grosjo commented Jan 14, 2025

reox commented Jan 15, 2025

grosjo commented Feb 2, 2025

grosjo commented Feb 22, 2025

Package version stuck on 1.5.5-1 on debian 12 #178

Package version stuck on 1.5.5-1 on debian 12 #178

Comments

seb1k commented Dec 1, 2024

grosjo commented Dec 1, 2024

jnahmias commented Dec 9, 2024

adixp2010 commented Dec 9, 2024 • edited Loading

seb1k commented Dec 9, 2024 • edited Loading

seb1k commented Dec 9, 2024 • edited Loading

grosjo commented Dec 10, 2024

seb1k commented Dec 10, 2024

grosjo commented Dec 10, 2024

seb1k commented Dec 10, 2024 • edited Loading

grosjo commented Dec 11, 2024

seb1k commented Dec 11, 2024

grosjo commented Dec 11, 2024

seb1k commented Dec 11, 2024

grosjo commented Dec 29, 2024

grosjo commented Dec 29, 2024

grosjo commented Jan 2, 2025

seb1k commented Jan 7, 2025

grosjo commented Jan 9, 2025

grosjo commented Jan 14, 2025

reox commented Jan 15, 2025

grosjo commented Feb 2, 2025

grosjo commented Feb 22, 2025

adixp2010 commented Dec 9, 2024 •

edited

Loading

seb1k commented Dec 9, 2024 •

edited

Loading

seb1k commented Dec 9, 2024 •

edited

Loading

seb1k commented Dec 10, 2024 •

edited

Loading