feat: send pre-message on messages with large attachments #7410

Simon-Laux · 2025-11-06T16:51:42Z

part of #7367

progress / what's to do:

send pre-message
The message's state must be set to MessageState::OutDelivered only after both messages are sent. If a read receipt is received, the message can be OutMdnRcvd or OutPending; let's just do whatever is easiest for now. Take care not to revert from OutMdnReceived to OutDelivered if we first receive a read receipt and then deliver the full message.
- this is already the case:
  - OutDelivered is set when a message is sent out and has no remaining send jobs in the smtp table for this message id - so already works since full message and pre message have same msgId in that table.
  - OutMdnRcvd is a "virtual" state ([Tracking Issue] Pre-Messages #7367 (comment)), so going back to OutDelivered can't happen anymore
delimit ChatFullMessageId with < and > like the other message ids
add tests
- test that pre message is sent for attachment larger than X
  - test that correct headers are present on both messages
  - also test that Autocrypt-gossip and selfavatar should never go into full-messages
- test that no pre message is sent for attachment smaller than X
  - no "is full message" header should be present
- test that pre message is not send for large webxdc update or large text
fix test receive_imf::receive_imf_tests::test_dont_reverify_by_self_on_outgoing_msg

src/chat.rs

and modify it to employ message reordering instead of a partially downloaded instance

full-messages and fix exclusion of gossip header, which was broken

`receive_imf::receive_imf_tests::test_dont_reverify_by_self_on_outgoing_msg`

easier solution for `should_do_gossip` with less changes. thanks to iequidoo for suggesting it apply test naming suggestion remove autoxrypt and self avatar check in `test_sending_pre_message` because it is already in `test_pre_message_contains_selfavatar_and_gossip_and_full_message_does_not` set self.attach_selfavatar to false in render method if pre message mode is full message merge `test_not_sending_pre_message_for_large_text` into `test_not_sending_pre_message_no_attachment`

Hocuri

I finally got around to review, sorry it took so long!

LGTM once comments are addressed.

src/download.rs

Hocuri · 2025-11-21T20:12:24Z

src/download.rs

+            .await;
+
+        let mut msg = Message::new(Viewtype::File);
+        msg.set_file_from_bytes(&alice.ctx, "test.bin", &[0u8; 300_000], None)?;


Just a heads-up: AFAIK, &[0u8; 300_000] actually includes 300kb of bytes into the test binary, which slows down compile times a bit. I'm not sure right now what could be a good fix for this; maybe it's fine. Maybe vec![300_000] would be better. But we can just ignore the problem for now; if we want to improve compile times, then doing something about the many include_str!()s in tests has likely the higher impact.

src/download.rs

src/headerdef.rs

src/chat.rs

src/download.rs

src/headerdef.rs

Co-authored-by: Hocuri <hocuri@gmx.de>

src/mimefactory.rs

iequidoo · 2025-11-24T06:08:18Z

src/mimefactory.rs

        Ok(message)
    }
+
+    pub fn will_be_encrypted(&self) -> bool {


Maybe rename to should_encrypt()? It sounds more consistent with other should_* functions like should_skip_autocrypt()

I think "should" sounds to vague in this case. it is used to set a variable with the name is_encrypted.

Co-authored-by: iequidoo <117991069+iequidoo@users.noreply.github.com>

Part of #7367 - Remove partial downloads (remove creation of the stub messages) (#7373) - Remove "Download maximum available until" and remove stock string `DC_STR_DOWNLOAD_AVAILABILITY` (#7369) - Send pre-message on messages with large attachments (#7410) - Pre messages can now get read receipts (#7433) progress / what's to do: - [x] send pre-message - [x] The message's state must be set to MessageState::OutDelivered only after both messages are sent. If a read receipt is received, the message can be OutMdnRcvd or OutPending; let's just do whatever is easiest for now. Take care not to revert from OutMdnReceived to OutDelivered if we first receive a read receipt and then deliver the full message. - this is already the case: - `OutDelivered` is set when a message is sent out and has no remaining send jobs in the smtp table for this message id - so already works since full message and pre message have same msgId in that table. - `OutMdnRcvd` is a "virtual" state (#7367 (comment)), so going back to `OutDelivered` can't happen anymore - [x] delimit `ChatFullMessageId` with `<` and `>` like the other message ids - [x] add tests - [x] test that pre message is sent for attachment larger than X - test that correct headers are present on both messages - also test that Autocrypt-gossip and selfavatar should never go into full-messages - [x] test that no pre message is sent for attachment smaller than X - no "is full message" header should be present - [x] test that pre message is not send for large webxdc update or large text - [x] fix test `receive_imf::receive_imf_tests::test_dont_reverify_by_self_on_outgoing_msg`

Part of #7367 - Remove partial downloads (remove creation of the stub messages) (#7373) - Remove "Download maximum available until" and remove stock string `DC_STR_DOWNLOAD_AVAILABILITY` (#7369) - Send pre-message on messages with large attachments (#7410) - Pre messages can now get read receipts (#7433) progress / what's to do: - [x] send pre-message - [x] The message's state must be set to MessageState::OutDelivered only after both messages are sent. If a read receipt is received, the message can be OutMdnRcvd or OutPending; let's just do whatever is easiest for now. Take care not to revert from OutMdnReceived to OutDelivered if we first receive a read receipt and then deliver the full message. - this is already the case: - `OutDelivered` is set when a message is sent out and has no remaining send jobs in the smtp table for this message id - so already works since full message and pre message have same msgId in that table. - `OutMdnRcvd` is a "virtual" state (#7367 (comment)), so going back to `OutDelivered` can't happen anymore - [x] delimit `ChatFullMessageId` with `<` and `>` like the other message ids - [x] add tests - [x] test that pre message is sent for attachment larger than X - test that correct headers are present on both messages - also test that Autocrypt-gossip and selfavatar should never go into full-messages - [x] test that no pre message is sent for attachment smaller than X - no "is full message" header should be present - [x] test that pre message is not send for large webxdc update or large text - [x] fix test `receive_imf::receive_imf_tests::test_dont_reverify_by_self_on_outgoing_msg` feat: receive pre-messages and adapt download on demand fix python lint errors receive pre-mesages, start with changes to imap loop. refactor: move download code from `scheduler.rs` to `download.rs`, also move `get_msg_id_by_rfc724_mid` to `MsgId::get_by_rfc724_mid` `MAX_FETCH_MSG_SIZE` is no longer unused Parse if it is a pre-message or full-message start with receiving logic get rid of `MsgId::get_by_rfc724_mid` because it was a duplicate of `message::rfc724_mid_exists` docs: add hint to `MimeMessage::from_bytes` stating that it has side-effects. receiving full message send and receive `attachment_size` and set viewtype to text in pre_message metadata as struct in pre-message in header. And fill params that we can already fill from the metadata. Also add a new api to check what viewtype the message will have once downloaded. api: jsonrpc: add `full_message_view_type` to `Message` and `MessageInfo` make PreMsgMetadata.to_header_value not consume self/PreMsgMetadata add api to merge params on download full message: merge new params into old params and remove full-message metadata params move tests to `src/tests/pre_messages.rs` dynamically allocate test attachment bytes fix detection of pre-messages. (it looked for the ChatFullMessageId header in the unencrypted headers before) fix setting dl state to avaiable on pre-messages fix: save pre message with rfc724_mid of full message als disable replacement for full messages add some receiving tests and update test todo for premessage metadata test: process full message before pre-message test receive normal message some serialization tests for PreMsgMetadata remove outdated todo comment test that pre-message contains message text PreMsgMetadata: test_build_from_file_msg and test_build_from_file_msg test: test_receive_pre_message_image Test receiving the full message after receiving an edit after receiving the pre-message test_reaction_on_pre_message test_full_download_after_trashed test_webxdc_update_for_not_downloaded_instance simplify fake webxdc generation in test_webxdc_update_for_not_downloaded_instance test_markseen_pre_msg test_pre_msg_can_start_chat and test_full_msg_can_start_chat test_download_later_keeps_message_order test_chatlist_event_on_full_msg_download fix download not working log splitting into pre-message add pre-message info to text when loading from db. this can be disabled with config key `hide_pre_message_metadata_text` if ui wants to display it in a prettier way. update `download_limit` documentation more logging: log size of pre and post messages rename full message to Post-Message split up the pre-message tests into multiple files dedup test code by extracting code to create test messages into util methods remove post_message_view_type from api, now it is only used internally for tests remove `hide_pre_message_metadata_text` config option, as there currently is no way to get the full message viewtype anymore Update src/download.rs resolve comment use `parse_message_id` instead of removing `<>`parenthesis it manually fix available_post_msgs gets no entries handle forwarding and add a test for it. convert comment to log warning event on unexpected download failure add doc comment to `simple_imap_loop` more logging handle saving pre-message to self messages and test. refactor: Simplify the logic for downloading a bit Immediately fully download all messages that do not have the `Chat-Is-Post-Message` header. This way, we simplify the logic for when which messages are downloaded, there are no differences at all anymore between 'background_fetch' and 'normal fetch'. Also, we prevent message reordering when reveiving a message from a legacy client. Messages larger than 1MB without `Chat-Is-Post-Message` should be rare, so, we do not expect this to really worsen things. Update deltachat-ffi/deltachat.h less .clone() clippy fixes rustfmt restore test_download_on_demand use anyhow remove TODO remove PartialDownloadMsgBody Add DC_STR_CONTACT stock string remove stock string TODOs remove optional todo feat: Don't put text into post-message This way, there will be no duplicate text if a chat partner is using an older DC version. If the chat partner is using a new DC version, then this will have no effect. fix: Make pre-messages db migration backwards-compatible See this comment that described the problem: https://github.com/chatmail/core/pull/7371/files#r2612182387 > The migration removes the `msg_id` column from the `download` table, so that, if you import the backup into an older DC version, you get the error `Failed inbox fetch_idle: Failed to download messages: no such column: msg_id in SELECT msg_id FROM download at offset 7: Error code 1: SQL error or missing database.` > > So, we need: > - either to make sure that things somewhat work if the user downgrades > - or to again increase `backup_version`, and then again coordinate that releases are done on all platforms simultaneously (which I would like to avoid). This PR here implements the first solution: Re-add `download.msg_id`, so that an older core can handle the database scheme. Co-authored-by: Hocuri <hocuri@gmx.de>

- Remove partial downloads (remove creation of the stub messages) (#7373) - Remove "Download maximum available until" and remove stock string `DC_STR_DOWNLOAD_AVAILABILITY` (#7369) - Send pre-message on messages with large attachments (#7410) - Pre messages can now get read receipts (#7433) Co-authored-by: Hocuri <hocuri@gmx.de>

- Remove partial downloads (remove creation of the stub messages) (#7373) - Remove "Download maximum available until" and remove stock string `DC_STR_DOWNLOAD_AVAILABILITY` (#7369) - Send pre-message on messages with large attachments (#7410) - Pre messages can now get read receipts (#7433) Co-authored-by: Hocuri <hocuri@gmx.de> Co-authored-by: iequidoo <dgreshilov@gmail.com>

Simon-Laux self-assigned this Nov 6, 2025

Simon-Laux commented Nov 6, 2025

View reviewed changes

src/chat.rs Outdated Show resolved Hide resolved

Simon-Laux mentioned this pull request Nov 6, 2025

[Tracking Issue] Pre-Messages #7367

Closed

7 tasks

Simon-Laux force-pushed the simon/sending-pre-message branch 2 times, most recently from a283803 to fcdd28c Compare November 8, 2025 18:38

Simon-Laux mentioned this pull request Nov 8, 2025

feat: pre-messages / next version of download on demand #7371

Merged

9 tasks

Simon-Laux marked this pull request as ready for review November 8, 2025 20:23

Hocuri mentioned this pull request Nov 11, 2025

remove: partial downloads (remove creation of the stub messages) #7373

Merged

2 tasks

Simon-Laux force-pushed the simon/remove-partial-download-of-messages branch from e564a9b to 05a1967 Compare November 12, 2025 19:38

Simon-Laux requested review from Hocuri, iequidoo and link2xt November 14, 2025 21:25

Simon-Laux force-pushed the simon/sending-pre-message branch from 90b61a1 to 1f0cc45 Compare November 14, 2025 21:35

Base automatically changed from simon/remove-partial-download-of-messages to pre-messages November 14, 2025 22:52

Simon-Laux force-pushed the simon/sending-pre-message branch from e97a448 to 2b900c3 Compare November 14, 2025 22:59

Simon-Laux and others added 14 commits November 16, 2025 21:07

fix python lint errors

ba573bb

fix lint of python test

b69082d

restore test test_webxdc_update_for_not_downloaded_instance and rename

17df902

and modify it to employ message reordering instead of a partially downloaded instance

remove test_webxdc_update_for_not_yet_received_instance

6f9823f

feat: send pre-message on messages with large attachments

6042fd0

allow unused for MAX_FETCH_MSG_SIZE

1954506

remove todo comment

f49ecfa

add tests for sending pre-messages

bf5d081

delimit Chat-Full-Message-ID with <> like other message ids.

709a965

fix test

9ffe300

test that Autocrypt-gossip and selfavatar should never go into

a3846ee

full-messages and fix exclusion of gossip header, which was broken

fix bug that broke

74a1fc9

`receive_imf::receive_imf_tests::test_dont_reverify_by_self_on_outgoing_msg`

cargo fmt after rebase

2d32a58

fix typo

305498d

Simon-Laux force-pushed the simon/sending-pre-message branch from 2b900c3 to 305498d Compare November 16, 2025 20:07

Simon-Laux requested a review from iequidoo November 23, 2025 02:28

Hocuri reviewed Nov 23, 2025

View reviewed changes

Simon-Laux added 3 commits November 24, 2025 03:01

reference to test contexts

027f3d9

assert!(! to assert_eq!(,false for "easier" code reading

3a36854

test: simplify chat creation by using .create_chat_with_contact

5ec079c

Simon-Laux commented Nov 24, 2025

View reviewed changes

src/headerdef.rs Outdated Show resolved Hide resolved

Simon-Laux and others added 5 commits November 24, 2025 02:22

Apply suggestions from code review

d5f8819

Co-authored-by: Hocuri <hocuri@gmx.de>

remove now unessesary test

3cc7a56

sql style

2ef8b6f

rm unused imports

413a6ca

tests: .headers instead of .get_headers()

9d3a506

iequidoo reviewed Nov 24, 2025

View reviewed changes

src/mimefactory.rs Outdated Show resolved Hide resolved

src/mimefactory.rs Show resolved Hide resolved

iequidoo reviewed Nov 24, 2025

View reviewed changes

iequidoo approved these changes Nov 24, 2025

View reviewed changes

Hocuri approved these changes Nov 24, 2025

View reviewed changes

Simon-Laux and others added 3 commits November 26, 2025 14:51

get_smtp_rows_for_msg now returns SentMessage instead of tuple

1dcd328

use bob.parse_msg(msg) instead of using mimeparser directly

8737c29

Update src/mimefactory.rs

381b709

Co-authored-by: iequidoo <117991069+iequidoo@users.noreply.github.com>

Simon-Laux merged commit 1ec5606 into pre-messages Nov 26, 2025
17 of 19 checks passed

Simon-Laux deleted the simon/sending-pre-message branch November 26, 2025 14:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: send pre-message on messages with large attachments #7410

feat: send pre-message on messages with large attachments #7410

Uh oh!

Simon-Laux commented Nov 6, 2025 •

edited

Loading

Uh oh!

Uh oh!

Hocuri left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Hocuri Nov 21, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

iequidoo Nov 24, 2025

Uh oh!

Simon-Laux Nov 26, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

feat: send pre-message on messages with large attachments #7410

feat: send pre-message on messages with large attachments #7410

Uh oh!

Conversation

Simon-Laux commented Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Hocuri left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Hocuri Nov 21, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

iequidoo Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

Simon-Laux Nov 26, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Simon-Laux commented Nov 6, 2025 •

edited

Loading

Hocuri left a comment •

edited

Loading