[Dovecot/FTS] Replace Solr with Xapian #4456

DerLinkman · 2022-02-07T07:41:06Z

DO NOT MERGE UNTIL ALL CHECKS ARE MADE

This PR replaces Solr FTS with Xapian FTS

It is usable but not nearly finished!

What is done? What needs to be added?:

Xapian itself

Dovecot Update + Xapian compilation
Xapian variable replacement from mailcow.conf and docker-compose.yml
Xapian Replacement in generate-config.sh & update.sh
Addition of new extra config file to control the Xapian Plugin

WebUI Stuff

Reorganisation of the general UI (Indevelopment by @FreddleSpl0it)

NOT Final | Revision 2
Xapian Replacement in WebUI
Add new Reindex Dovecot Indexes Button in WebUI (controlled with DockerAPI) (Xapian section) NOT FINAL

Behind the scenes stuff (API, Translations etc.)

Remove/Replace Solr Strings with Xapian (Translations)
Solr API removal (Xapian replacement?)
Cleanups (Dovecot Image)
Final Tests

This Push adds the bullseye rebase + the compilation of the XAPIAN Core + Plugin to run with Dovecot 2.3.17

First revision. Waiting on: grosjo/fts-xapian#115

This added a include try pointing on the file instead.

Jan(moo)uary Update 2022 - Revision A (2022-01a) (#4445)

* [Web] add github version tag * [Web] add github version tag * [Web] add github version tag * [Web] add github version tag * [Web] add github version tag * [Web] add github version tag error handling * [Web] add github version tag error handling * Passwordless SOGo auth: support for calendar invitations and calendar/contacts subscriptions Inviting someone to a calendar event triggers a request to /SOGo/so/otheruser@example.com/freebusy.ifb/ajaxRead. Subscribing to someone's calendar/contacts triggers a request to /SOGo/so/otheruser@example.com/foldersSearch. The email address in the URL is different from the logged-in user, which needs to be handled appropriately by sogo-auth.php. * [Web] add github version tag - adjust css * [Compose] Update SOGo Autoreply Schedule to 5m Based on the advice of inverse (SOGo developer). Thanks to https://github.com/jmber Closes: #4436 * [Web] add github version tag - move twig globals * [Web] add github version tag - missing </div> * Passwordless SOGo auth: improvements for when accessing other users * [WebAuthn] fido2 passwordless auth - fix (#4440) * [WebAuthn] fido2 revert * [WebAuthn] set UV flags to 'discouraged' * [WebAuthn] revert - set UV flags to 'discouraged' * Update clamav to 0.104.2 * Update clamav to 0.104.2 * Update dovecot to 2.3.18 Update gosu to 1.14 Use debian bullseye as base * [Web] Updated lang.es.json [CI SKIP] (#4453) Co-authored-by: Fijxu <fijxu@zzls.xyz> Co-authored-by: milkmaker <milkmaker@mailcow.de> Co-authored-by: Fijxu <fijxu@zzls.xyz> Co-authored-by: FreddleSpl0it <patschul@posteo.de> Co-authored-by: FreddleSpl0it <75116288+FreddleSpl0it@users.noreply.github.com> Co-authored-by: Michael Kuron <mkuron@users.noreply.github.com> Co-authored-by: Peter <magic@kthx.at> Co-authored-by: milkmaker <milkmaker@mailcow.de> Co-authored-by: Fijxu <fijxu@zzls.xyz>

[XAPIAN] Added Solr Replacement [BETA]

MAGICCC · 2022-02-07T16:56:06Z

Nice! We could convert the PR into a draft so we won't merge it accidentally :D

Also we need to redo the translation parts (renaming Solr to Xapian)
What about old installations and their configs? Do you plan to try to remove the corresponding lines from the mailcow.conf?
E.g.
https://github.com/mailcow/mailcow-dockerized/blob/master/generate_config.sh#L250-L257

DerLinkman · 2022-02-07T17:09:28Z

Why do we need to remove the Xapian translation? It is replacing the solr. I didn't tested it until now. New mailcow Installations works flawlessly

mkuron

Could you please provide some rough numbers how much memory and disk space Xapian and Solr take up for an "average" user‘s mailbox, e.g. 10000 messages? Xapian should be better than Solr, but we should know by how much.

data/Dockerfiles/dovecot/Dockerfile

data/Dockerfiles/dovecot/decode2text.sh

andryyy · 2022-07-19T05:46:05Z

As long as this is just to play around, it’s fine. Don’t ever merge this to master, please, before several things are fixed and there is an official release.

VermiumSifell · 2023-01-18T09:20:19Z

I think the issue title should be named Replace Solr with Xapian, as I currently understands the title as going from Xapian to Solr

DerLinkman · 2023-01-18T11:15:36Z

I think the issue title should be named Replace Solr with Xapian, as I currently understands the title as going from Xapian to Solr

Absolutely right. Don't know if i was drunk or something why that happend :)

Thanks anyways

jpbaril · 2024-01-30T14:19:05Z

@DerLinkman Feature abandoned?

DerLinkman · 2024-01-30T14:28:44Z

No just redone with the current mailcow development branches (see: https://github.com/mailcow/mailcow-dockerized/tree/feat/fts-flatcurve)

ntimo and others added 24 commits January 21, 2022 21:29

[API] Fix minor issue in api docs

c82f38a

[GH-Actions][stale] Add neverstale label to exempt list

cfba96f

[Dovecot] Rebase on Bullseye + Xapian Compile

bca09e3

This Push adds the bullseye rebase + the compilation of the XAPIAN Core + Plugin to run with Dovecot 2.3.17

Merge branch 'mailcow:master' into Dovecot-FTS-XAPIAN

3a9177b

[Dovecot] Add decode2text.sh

3110077

[Update.sh] Replace Solr with Xapian

148b511

[Config] Replace Solr with Xapian (Remove Solr Binds)

26c5ed7

[Config] Readded default Value for Xapian Heap

f2e0e50

[Compose] Replace solr to Xapian (in dovecot)

df2c33d

First revision. Waiting on: grosjo/fts-xapian#115

[Compose] Remove solr from ipv6-nat dependencies

570170a

[Dovecot] Added Xapian default config

ed89414

[Dovecot] Changed Xapian default to 1024m instead of 2G

cdce97b

[Dovecot] Added seperate XAPIAN Conf

f09ca0a

Delete FTS-Xapian.conf

c98ef0d

[Dovecot] Added seperate XAPIAN Conf

99ea569

[Dovecot] Removed Xapian from dovecot.conf

30e2d94

This added a include try pointing on the file instead.

[Dovecot] Added Xapian include

60ddfe3

Update syslog-ng-redis_slave.conf

2d6ce92

Update syslog-ng.conf

d079ff4

[Dovecot] Added Xapian Parameters from mailcow.conf

3d637ac

[Compose] Changed Ofelia CMD for fts optimize

28582c5

Merge pull request #2 from mailcow/master

db73f83

Jan(moo)uary Update 2022 - Revision A (2022-01a) (#4445)

Merge pull request #4455 from DerLinkman/Dovecot-FTS-XAPIAN

aec2dd1

[XAPIAN] Added Solr Replacement [BETA]

DerLinkman marked this pull request as draft February 7, 2022 17:10

mkuron requested changes Feb 7, 2022

View reviewed changes

data/Dockerfiles/dovecot/Dockerfile Outdated Show resolved Hide resolved

data/Dockerfiles/dovecot/Dockerfile Outdated Show resolved Hide resolved

data/Dockerfiles/dovecot/decode2text.sh Outdated Show resolved Hide resolved

DerLinkman added 2 commits February 23, 2022 09:45

Cleanup Dockerfile + Added new dependencies

c0e6625

Delete decode2text.sh

5bd3394

moo added 2 commits April 29, 2022 14:28

Changed Build process to fts-flatcurve Version of Xapian FTS

bfa81b3

Renamed, reconfigured needed Xapian files for Flatcurve

3425bcf

Base automatically changed from staging to master May 5, 2022 15:30

Restored manual Dovecot Build

f0e2731

FreddleSpl0it mentioned this pull request Jun 8, 2022

Dashboard showing quotas #4622

Open

DerLinkman added 3 commits July 18, 2022 11:15

Removed Solr Leftovers + renamed flatcurve conf

ed476aa

Using Stable Dovecot Builds for flatcurve

8607942

Multi Build Dovecot Building + Config Optimizations

df3de09

DerLinkman added 7 commits July 20, 2022 15:49

Enabled Substring Search for flatcurve

b0fd978

Updated Dovecot Configs to be up to date

3ba3f1c

Updated newer Compenent Versions

04f55fc

Merge branch 'staging' into feature/fts-xapian

59a4031

Merge remote-tracking branch 'origin/staging' into feature/fts-xapian

60871fa

Added 1.20-Xapian Image

a0ae59f

Implemented current state of UI (BS5)

eb4a33d

DerLinkman added this to the 2023 milestone Nov 21, 2022

Merge branch 'feature/bootstrap5' into feature/fts-xapian

0408bbf

DerLinkman changed the base branch from master to staging January 18, 2023 11:14

VermiumSifell changed the title ~~[Dovecot/FTS] Replace Xapian with Solr~~ [Dovecot/FTS] Replace Solr with Xapian Jan 20, 2023

Merge branch 'master' into feature/fts-xapian

f09c853

DerLinkman modified the milestones: 2023, 2024 Jan 10, 2024

DerLinkman closed this Jan 30, 2024

DerLinkman deleted the feature/fts-xapian branch January 30, 2024 10:37

DerLinkman mentioned this pull request Jan 30, 2024

dovecot: replace solr fts with flatcurve (xapian) #5680

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Dovecot/FTS] Replace Solr with Xapian #4456

[Dovecot/FTS] Replace Solr with Xapian #4456

DerLinkman commented Feb 7, 2022 •

edited

Loading

MAGICCC commented Feb 7, 2022 •

edited

Loading

DerLinkman commented Feb 7, 2022

mkuron left a comment

andryyy commented Jul 19, 2022

VermiumSifell commented Jan 18, 2023

DerLinkman commented Jan 18, 2023

jpbaril commented Jan 30, 2024 •

edited

Loading

DerLinkman commented Jan 30, 2024

[Dovecot/FTS] Replace Solr with Xapian #4456

[Dovecot/FTS] Replace Solr with Xapian #4456

Conversation

DerLinkman commented Feb 7, 2022 • edited Loading

DO NOT MERGE UNTIL ALL CHECKS ARE MADE

Xapian itself

WebUI Stuff

Behind the scenes stuff (API, Translations etc.)

MAGICCC commented Feb 7, 2022 • edited Loading

DerLinkman commented Feb 7, 2022

mkuron left a comment

Choose a reason for hiding this comment

andryyy commented Jul 19, 2022

VermiumSifell commented Jan 18, 2023

DerLinkman commented Jan 18, 2023

jpbaril commented Jan 30, 2024 • edited Loading

DerLinkman commented Jan 30, 2024

DerLinkman commented Feb 7, 2022 •

edited

Loading

MAGICCC commented Feb 7, 2022 •

edited

Loading

jpbaril commented Jan 30, 2024 •

edited

Loading