Recursive search in imap account

[egoitzau]

Hi!,

I have been looking at Z-push code in terms of searching for the imap (really combined) backend.

The ContentParameters class, could seem to have a method that returns true in order to make possible mailbox wide search. This method should be called GetSearchDeepTraversal.

I assume it's not created nowadays because it imap_open each folder in the mailbox and performs an imap_search then. That could be slow and could end up in timeouts in several different elements. Am I wrong?.

Perhaps here it would be nice to have a way of searching like the Roundcube has, which uses threading and concurrency for finding in several mailboxes at the same time. Am I wrong?.

I'm trying to identify the problem and to see if you could help the project in some manner, but for that, we need to know the exact limitation with searches in this backend, although it should be something similar to the commented. Could you please confirm me it?.

Cheers!!

[matidau]

Hi @egoitzau

I haven't dug too deeply in to the Imap search, apart from adding an FTS engine to dovecot reduces timeouts.

I can't confirm either way, without researching and testing it myself.

Cheers,
Mat

[egoitzau]

Hi @matidau !

Imap searches seem to require a select for the mailbox (folder) and a search operation.

It seems that at least in Android, as folder is not specified INBOX is used... (in the imap_open() stage for later doing the imap_search()).

I saw in the code, that a if checked if that function returned true, which nowadays is impossible to happen due to not existing for that class.

Could you know why the line 2041 at backend/imap/imap.php checks for true?. Do you recognize that code?. It's written by you or...

Cheers!

[matidau]

Is 2040 the line that you are referring to?

Z-Push/src/backend/imap/imap.php

Line 2040 in 2c11075

if ($cpo->GetSearchDeepTraversal()) { // Recursive search

I didn't write this and am not familiar with it. You can use blame on Github to see who committed the code.

[egoitzau]

yep! could be!

I'm going to do some attempts...

Cheers!

[egoitzau]

Hi!

I have been working on improving searches mainly for Android envs with the Gmail app. As Opensync app has more or less dissapeared, we needed a replacement for that app. We had been using Z-push from some time ago, for some other purposes in iOS devices.

We use combined backend with imap and caldav. We are leaving the default Activesync settings (except the server) that Android leaves when you create an Exchange account for these Z-push accounts.

The relevant config (I suppose at least) elements are :

 //define('SEARCH_PROVIDER', '');
 define('SEARCH_WAIT', 600);
 define('SEARCH_MAXRESULTS', 5000);

With the following patch applied over 2.7.1 version, it seems to be working better with the Gmail app and Android 13. Searches seem to work at least far more better.

diff-zpush-android.zip

I wanted to share this patch in case it could help someone or could be used for improving search command behavior in Z-push and imap backends. Without it, it does not support searching in all folders and the results returned are always the first 10 ones.

Cheers!

[matidau]

Thanks for sharing @egoitzau !

Would you be willing to do the work to put this forward as a PR?

Or release it under the AGPL, so I or someone else can in the future, with the following words?

Released under the GNU Affero General Public License (AGPL), version 3 and Trademark Additional Terms.

Cheers,
Mat

[chittaranjan870]

Hi!

I have been working on improving searches mainly for Android envs with the Gmail app. As Opensync app has more or less dissapeared, we needed a replacement for that app. We had been using Z-push from some time ago, for some other purposes in iOS devices.

We use combined backend with imap and caldav. We are leaving the default Activesync settings (except the server) that Android leaves when you create an Exchange account for these Z-push accounts.

The relevant config (I suppose at least) elements are :

 //define('SEARCH_PROVIDER', '');
 define('SEARCH_WAIT', 600);
 define('SEARCH_MAXRESULTS', 5000);

With the following patch applied over 2.7.1 version, it seems to be working better with the Gmail app and Android 13. Searches seem to work at least far more better.

diff-zpush-android.zip

I wanted to share this patch in case it could help someone or could be used for improving search command behavior in Z-push and imap backends. Without it, it does not support searching in all folders and the results returned are always the first 10 ones.

Cheers!

Is their any solution? I have facing same searching issue in gmail app and below error is showing
[56928] [WARN] [ux1@domain.com] Searchtype 'MAILBOX' is not supported.
[56928] [ INFO] [ux1@domain.com] BackendLDAP->Logoff(): Disconnection successfull.
[56928] [ INFO] [ux1@domain.com] cmd='Search' memory='1.86 MiB/2.00 MiB' time='0.02s' devType='Android' devId='androidc1310712648' getUser='ux1@domain.com' from='192.168.10.152' idle='0s' version='2.7.4
' method='POST' httpcode='200'

[egoitzau]

Hi @matidau !

Really said we have made another little improvement too.
imap.php.diff.zip

For Android, the way the Gmail app works in at least Android 13, it just searched words a "TEXT" search with imap_search(). This is awful if you need to find something for instance between the 70 and the 80 result, because it shows from 10 to 10. By the way, it seems that when you hit "Load more" several times, I don't really know why but I bet for a malfunction in the Gmail app, it stops showing results.

For the previous paragraph reasons, we have improved slightly the search. It's very weird and uncommon to see someone trying to find as text patterns like subject:someword or from:someword. What they usually will want is to be searching for mails with that word in the subject or the from. So the attached patch in this comment does the trick. Apart from that, it makes easy to extend the search possibilities with other headers like the to header or whatever meanwhile imap_search() supports that keywords for searching.

Know, you can do a search like from:james car that means that you want to search a mail from James with the word car in anywhere. Same example for the subject.

About the fact of releasing, of course absolutely free. We just wanted to contribute this project the be improved and to encourage people to improve it and everyone to stand to benefit of this contributions done by the system admins/system engineers. What do you really need for this contribution to be possible?. A document saying it's opened under AGPL or... please tell me what is needed and we will do.

About the fact of releasing, and as secondary topic, I would be very thankful if you could mention my company (Sarenet) and just thanking for the contribution. As stated if could be possible.

Thank you so much,
Cheers!

[matidau]

If you state the words

Released under the GNU Affero General Public License (AGPL), version 3 and Trademark Additional Terms.

in a reply here that would be good enough for me.

If you wanted to put in the extra work, you can create a fork and a pull request. This will let you include the mention of Sarenet in the commit message or as a comment in the code. A guide for creating a fork and a pull request is here: https://docs.github.com/en/get-started/exploring-projects-on-github/contributing-to-a-project

Cheers,
Mat

[egoitzau]

Released under the GNU Affero General Public License (AGPL), version 3 and Trademark Additional Terms.

[egoitzau]

Is it ok? :)

Ok then... I'll do that pull request when proposing the following patch in order to be able to just name and thank to Sarenet the possibility of contributing to this project :)

Thanks Mat!!

If you need something else for being realeased or use the code... just let me know and we will carry out :)

[matidau]

Is it ok? :)

This is perfect.😊

[egoitzau]

Hi! Have you tried the commented patch?. I’m using Combined backend too which relies in imap finally… If you perform a search with a server side search capable MUA does it work?. Cheers! ============================= Egoitz Aurrekoetxea Departamento de sistemas 94 - 420 94 70 | ***@***.*** S A R E N E T S.A.U. Parque Tecnológico. Edificio 103 | 48170 Zamudio (Bizkaia) - www.sarenet.es Antes de imprimir este correo electrónico piense si es necesario hacerlo. De: chittaranjan870 ***@***.***> Enviado el: miércoles, 23 de octubre de 2024 14:16 Para: Z-Hub/Z-Push ***@***.***> CC: egoitzau ***@***.***>; Mention ***@***.***> Asunto: Re: [Z-Hub/Z-Push] Recursive search in imap account (Discussion #48) Hi! I have been working on improving searches mainly for Android envs with the Gmail app. As Opensync app has more or less dissapeared, we needed a replacement for that app. We had been using Z-push from some time ago, for some other purposes in iOS devices. We use combined backend with imap and caldav. We are leaving the default Activesync settings (except the server) that Android leaves when you create an Exchange account for these Z-push accounts. The relevant config (I suppose at least) elements are : //define('SEARCH_PROVIDER', ''); define('SEARCH_WAIT', 600); define('SEARCH_MAXRESULTS', 5000); With the following patch applied over 2.7.1 version, it seems to be working better with the Gmail app and Android 13. Searches seem to work at least far more better. diff-zpush-android.zip <https://github.com/Z-Hub/Z-Push/files/14048756/diff-zpush-android.zip> I wanted to share this patch in case it could help someone or could be used for improving search command behavior in Z-push and imap backends. Without it, it does not support searching in all folders and the results returned are always the first 10 ones. Cheers! Is their any solution? I have facing same searching issue in gmail app and below error is showing [56928] [WARN] ***@***.*** ***@***.***> ] Searchtype 'MAILBOX' is not supported. [56928] [ INFO] ***@***.*** ***@***.***> ] BackendLDAP->Logoff(): Disconnection successfull. [56928] [ INFO] ***@***.*** ***@***.***> ] cmd='Search' memory='1.86 MiB/2.00 MiB' time='0.02s' devType='Android' devId='androidc1310712648' ***@***.*** ***@***.***> ' from='192.168.10.152' idle='0s' version='2.7.4 ' method='POST' httpcode='200' — Reply to this email directly, view it on GitHub <#48 (reply in thread)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABPM73M6MJSCII43XS3KXMDZ46HPPAVCNFSM6AAAAABQOVUQBSVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCMBSHA3DQOA> . You are receiving this because you were mentioned. <https://github.com/notifications/beacon/ABPM73O4XAR45L7R5BVQ5EDZ46HPPA5CNFSM6AAAAABQOVUQBSWGG33NNVSW45C7OR4XAZNRIRUXGY3VONZWS33OINXW23LFNZ2KUY3PNVWWK3TUL5UWJTQAVBENA.gif> Message ID: ***@***.*** ***@***.***> >

[chittaranjan870]

searching issue in gmail apps now working after set the define('SEARCH_PROVIDER', ''); to blank in z-push main configuration file.

Thanks