Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sync and Send NOT reusing connections for operations #10039

Open
SebastianKrupinski opened this issue Aug 21, 2024 · 3 comments
Open

Sync and Send NOT reusing connections for operations #10039

SebastianKrupinski opened this issue Aug 21, 2024 · 3 comments
Assignees
@SebastianKrupinski
Labels
0. to triage bug

Comments

@SebastianKrupinski
Copy link
Contributor

Steps to reproduce

  1. Send or Sync messages
  2. Check server or mail client logs

Expected behavior

Use a single connection for all operations then close connection once completed

Actual behavior

Sync operation opens new IMAP connection for every mailbox that needs to sync.

Send operation opens one new SMTP connection and two new IMAP connections for every email it sends.

Mail app version

Lastest Dev

Mailserver or service

IMAP/SMTP

Operating system

No response

PHP engine version

PHP 8.3

Web server

Nginx

Database

MySQL

Additional info

No response
@SebastianKrupinski
Copy link
Contributor Author

SebastianKrupinski commented Aug 21, 2024 •
edited
Loading

Test Results of sending 3 messages at the same time from the same account.

Thu, 08 Aug 2024 12:37:22 - INFO - Network connection ended (network.connection-end) listenerId = "submission", protocol = smtp, remoteIp = 127.0.0.1, remotePort = 60796, elapsed = 29ms
Thu, 08 Aug 2024 12:37:22 - INFO - SMTP EHLO command (smtp.ehlo) listenerId = "submission", protocol = smtp, remoteIp = 127.0.0.1, remotePort = 60796, domain = "localhost"
Thu, 08 Aug 2024 12:37:22 - INFO - Network connection started (network.connection-start) listenerId = "submission", protocol = smtp, remoteIp = 127.0.0.1, remotePort = 60796

Thu, 08 Aug 2024 12:37:20 - INFO - Network connection ended (network.connection-end) listenerId = "submission", protocol = smtp, remoteIp = 127.0.0.1, remotePort = 60792, elapsed = 28ms
Thu, 08 Aug 2024 12:37:20 - INFO - SMTP EHLO command (smtp.ehlo) listenerId = "submission", protocol = smtp, remoteIp = 127.0.0.1, remotePort = 60792, domain = "localhost"
Thu, 08 Aug 2024 12:37:20 - INFO - Network connection started (network.connection-start) listenerId = "submission", protocol = smtp, remoteIp = 127.0.0.1, remotePort = 60792

Thu, 08 Aug 2024 12:37:17 - INFO - Network connection ended (network.connection-end) listenerId = "submission", protocol = smtp, remoteIp = 127.0.0.1, remotePort = 58302, elapsed = 55ms
Thu, 08 Aug 2024 12:37:17 - INFO - SMTP EHLO command (smtp.ehlo) listenerId = "submission", protocol = smtp, remoteIp = 127.0.0.1, remotePort = 58302, domain = "localhost"
Thu, 08 Aug 2024 12:37:17 - INFO - Network connection started (network.connection-start) listenerId = "submission", protocol = smtp, remoteIp = 127.0.0.1, remotePort = 58302



Thu, 08 Aug 2024 12:43:50 - INFO - Network connection ended (network.connection-end) listenerId = "imap", protocol = imap, remoteIp = 127.0.0.1, remotePort = 53268, elapsed = 10ms
Thu, 08 Aug 2024 12:43:50 - INFO - Authentication successful (auth.success) listenerId = "imap", protocol = imap, remoteIp = 127.0.0.1, remotePort = 53268, name = "user1@testmail.com", accountId = 61, type = Individual
Thu, 08 Aug 2024 12:43:50 - INFO - Network connection started (network.connection-start) listenerId = "imap", protocol = imap, remoteIp = 127.0.0.1, remotePort = 53268

Thu, 08 Aug 2024 12:43:50 - INFO - Network connection ended (network.connection-end) listenerId = "imap", protocol = imap, remoteIp = 127.0.0.1, remotePort = 53262, elapsed = 47ms
Thu, 08 Aug 2024 12:43:50 - INFO - Authentication successful (auth.success) listenerId = "imap", protocol = imap, remoteIp = 127.0.0.1, remotePort = 53262, name = "user1@testmail.com", accountId = 61, type = Individual
Thu, 08 Aug 2024 12:43:50 - INFO - Network connection started (network.connection-start) listenerId = "imap", protocol = imap, remoteIp = 127.0.0.1, remotePort = 53262

Thu, 08 Aug 2024 12:43:50 - INFO - Network connection ended (network.connection-end) listenerId = "imap", protocol = imap, remoteIp = 127.0.0.1, remotePort = 53256, elapsed = 24ms
Thu, 08 Aug 2024 12:43:50 - INFO - Authentication successful (auth.success) listenerId = "imap", protocol = imap, remoteIp = 127.0.0.1, remotePort = 53256, name = "user1@testmail.com", accountId = 61, type = Individual
Thu, 08 Aug 2024 12:43:50 - INFO - Network connection started (network.connection-start) listenerId = "imap", protocol = imap, remoteIp = 127.0.0.1, remotePort = 53256

Thu, 08 Aug 2024 12:43:50 - INFO - Network connection ended (network.connection-end) listenerId = "imap", protocol = imap, remoteIp = 127.0.0.1, remotePort = 53252, elapsed = 27ms
Thu, 08 Aug 2024 12:43:50 - INFO - Authentication successful (auth.success) listenerId = "imap", protocol = imap, remoteIp = 127.0.0.1, remotePort = 53252, name = "user1@testmail.com", accountId = 61, type = Individual
Thu, 08 Aug 2024 12:43:50 - INFO - Network connection started (network.connection-start) listenerId = "imap", protocol = imap, remoteIp = 127.0.0.1, remotePort = 53252

Thu, 08 Aug 2024 12:43:50 - INFO - Network connection ended (network.connection-end) listenerId = "imap", protocol = imap, remoteIp = 127.0.0.1, remotePort = 53236, elapsed = 27ms
Thu, 08 Aug 2024 12:43:50 - INFO - Authentication successful (auth.success) listenerId = "imap", protocol = imap, remoteIp = 127.0.0.1, remotePort = 53236, name = "user1@testmail.com", accountId = 61, type = Individual
Thu, 08 Aug 2024 12:43:50 - INFO - Network connection started (network.connection-start) listenerId = "imap", protocol = imap, remoteIp = 127.0.0.1, remotePort = 53236

Thu, 08 Aug 2024 12:43:50 - INFO - Network connection ended (network.connection-end) listenerId = "imap", protocol = imap, remoteIp = 127.0.0.1, remotePort = 53222, elapsed = 68ms
Thu, 08 Aug 2024 12:43:50 - INFO - Authentication successful (auth.success) listenerId = "imap", protocol = imap, remoteIp = 127.0.0.1, remotePort = 53222, name = "user1@testmail.com", accountId = 61, type = Individual
Thu, 08 Aug 2024 12:43:50 - INFO - Network connection started (network.connection-start) listenerId = "imap", protocol = imap, remoteIp = 127.0.0.1, remotePort = 53222

Total of 3 SMTP connections created and 6 IMAP connections created.

Issue:
Ths Send chain modules, each module creates a new connection for each message instead of reusing the connection. https://github.com/nextcloud/mail/tree/main/lib/Send

Solutions:

  • Rewrite modules to pass IMAP and SMTP connection down the chain.
  • Rewrite connection factory method to cache connections based on hash of server host and username, and return cached connection on second call

@SebastianKrupinski
Copy link
Contributor Author

SebastianKrupinski commented Aug 21, 2024 •
edited
Loading

Test results of syncing an account, with 3 mailboxes (folders)

Thu, 08 Aug 2024 14:32:24 - INFO - Network connection ended (network.connection-end) listenerId = "imap", protocol = imap, remoteIp = 127.0.0.1, remotePort = 49226, elapsed = 38ms
Thu, 08 Aug 2024 14:32:24 - INFO - Authentication successful (auth.success) listenerId = "imap", protocol = imap, remoteIp = 127.0.0.1, remotePort = 49226, name = "user1@testmail.com", accountId = 61, type = Individual
Thu, 08 Aug 2024 14:32:24 - INFO - Network connection started (network.connection-start) listenerId = "imap", protocol = imap, remoteIp = 127.0.0.1, remotePort = 49226

Thu, 08 Aug 2024 14:32:24 - INFO - Network connection ended (network.connection-end) listenerId = "imap", protocol = imap, remoteIp = 127.0.0.1, remotePort = 49212, elapsed = 52ms
Thu, 08 Aug 2024 14:32:24 - INFO - Authentication successful (auth.success) listenerId = "imap", protocol = imap, remoteIp = 127.0.0.1, remotePort = 49212, name = "user1@testmail.com", accountId = 61, type = Individual
Thu, 08 Aug 2024 14:32:24 - INFO - Network connection started (network.connection-start) listenerId = "imap", protocol = imap, remoteIp = 127.0.0.1, remotePort = 49212

Thu, 08 Aug 2024 14:32:24 - INFO - Network connection ended (network.connection-end) listenerId = "imap", protocol = imap, remoteIp = 127.0.0.1, remotePort = 49210, elapsed = 98ms
Thu, 08 Aug 2024 14:32:24 - INFO - Authentication successful (auth.success) listenerId = "imap", protocol = imap, remoteIp = 127.0.0.1, remotePort = 49210, name = "user1@testmail.com", accountId = 61, type = Individual
Thu, 08 Aug 2024 14:32:24 - INFO - Network connection started (network.connection-start) listenerId = "imap", protocol = imap, remoteIp = 127.0.0.1, remotePort = 49210

Issue:
The connection is created in the mailbox sync function

mail/lib/Service/Sync/ImapToDbSynchronizer.php

Line 194 in a41a075

public function sync(Account $account,

instead of the Account Sync function

mail/lib/Service/Sync/ImapToDbSynchronizer.php

Line 103 in a41a075

public function syncAccount(Account $account,

which causes the connection to be created per mailbox.

Solutions:

  • Rewrite account sync function to create the connection and pass it to mailbox sync.
  • Rewrite connection factory method to cache connections based on hash of server host and username, and return cached connection on second call

@ChristophWurst
Copy link
Member

Rewrite connection factory method to cache connections based on hash of server host and username, and return cached connection on second call

We tried that and it caused too many open connections: #6174

@SebastianKrupinski SebastianKrupinski self-assigned this Aug 22, 2024
@SebastianKrupinski SebastianKrupinski mentioned this issue Sep 6, 2024
Draft
@meichthys meichthys mentioned this issue Oct 4, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@SebastianKrupinski SebastianKrupinski

Labels
0. to triage bug
Projects
💌 📅 👥 Groupware team
Status: No status
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ChristophWurst @SebastianKrupinski