remove: partial downloads (remove creation of the stub messages) #7373

Simon-Laux · 2025-10-30T16:40:22Z

remove code for partial downloads
remove tests for partial downloads and make a list of them so we can discuss if we need to re-add some in the new system or not
~~make download full message methods fail for now with not implemented error and comment what should happen later~~ doesn't need to change at this time (and possibly not at all)

~~I kept partial_download_msg_body stock string for now, as it may be reused in the new solution. because I made the linter ignore it, I added a to do item to #7367 to check if it was used in the end.~~ edit: outdated, the stock string was removed meanwhile.

removed partial download tests:

test_reaction_to_partially_fetched_msg -

core/deltachat-rpc-client/tests/test_something.py

Line 623 in 0d46626

def test_reaction_to_partially_fetched_msg(acfactory, tmp_path):
test_partial_download_and_ephemeral -

core/src/download.rs

Line 391 in 8b4c718

async fn test_partial_download_and_ephemeral() -> Result<()> {
test_status_update_expands_to_nothing -

core/src/download.rs

Line 425 in 8b4c718

async fn test_status_update_expands_to_nothing() -> Result<()> {
test_partial_download_trashed -

core/src/download.rs

Line 536 in 8b4c718

async fn test_partial_download_trashed() -> Result<()> {
test_partial_download_and_reaction -

core/src/reaction.rs

Line 929 in 8b4c718

async fn test_partial_download_and_reaction() -> Result<()> {
test_partial_group_consistency -

core/src/receive_imf/receive_imf_tests.rs

Line 4473 in 8b4c718

async fn test_partial_group_consistency() -> Result<()> {
test_partial_download_key_contact_lookup -

core/src/receive_imf/receive_imf_tests.rs

Line 5288 in 8b4c718

async fn test_partial_download_key_contact_lookup() -> Result<()> {
⭐test_webxdc_update_for_not_downloaded_instance -

core/src/webxdc/webxdc_tests.rs

Line 333 in 8b4c718

async fn test_webxdc_update_for_not_downloaded_instance() -> Result<()> {
test_no_partial_calls -

core/src/calls/calls_tests.rs

Line 617 in 8b4c718

async fn test_no_partial_calls() -> Result<()> {
test_markseen_not_downloaded_msg -

core/src/message/message_tests.rs

Line 331 in 8b4c718

async fn test_markseen_not_downloaded_msg() -> Result<()> {
test_msg_seen_on_imap_when_downloaded -

core/src/message/message_tests.rs

Line 404 in 8b4c718

async fn test_msg_seen_on_imap_when_downloaded() -> Result<()> {
test_create_group_with_big_msg -

core/src/receive_imf/receive_imf_tests.rs

Line 4398 in 8b4c718

async fn test_create_group_with_big_msg() -> Result<()> {
test_download_later -

core/src/receive_imf/receive_imf_tests.rs

Line 4333 in 8b4c718

async fn test_download_later() -> Result<()> {
test_prefer_references_to_downloaded_msgs -

core/src/receive_imf/receive_imf_tests.rs

Line 4792 in 8b4c718

async fn test_prefer_references_to_downloaded_msgs() -> Result<()> {
test_download_on_demand -

core/deltachat-rpc-client/tests/test_chatlist_events.py

Line 114 in 56370c2

def test_download_on_demand(acfactory: ACFactory) -> None:
test_download_limit_chat_assignment -

core/deltachat-rpc-client/tests/test_something.py

Line 723 in 56370c2

def test_download_limit_chat_assignment(acfactory, tmp_path, n_accounts):
test_webxdc_download_on_demand -

core/python/tests/test_1_online.py

Line 225 in 9bc2aee

def test_webxdc_download_on_demand(acfactory, data, lp):

part of #7367

Simon-Laux · 2025-10-30T20:13:35Z

run and fix python tests
- jsonrpc
- cffi
solve that it stops receiving messages (when used in desktop)

src/mimeparser.rs

Simon-Laux · 2025-10-31T18:52:22Z

src/receive_imf.rs

            info!(context, "Message already partly in DB, replacing.");
            Some(msg.chat_id)
+
+            // TODO: look at this place


todo comment

Note that there may be existing message-stubs that are not downloaded yet. After updating, the "Download" button will probably become non-functional; this is fine. We should make sure that DC doesn't crash when clicking "Download".

currently I'm still asking myself whether we have to to make this sacrifice or if we can still have a last version that has an option to download old message stubs and remove it in the version after...

As of the time of writing the download logic still works, because I have only removed the partial download logic that creates the stubs so far.

my current feeling/idea is to remove it in a dedicated pr as soon as it makes problems / the implementation of the new method harder - that way this pr would also be focused on the removal of just the stub creation part of partial download.

It's not that big of a sacrifice:

download-on-demand is just an option (i.e. many people don't have it activated, since they don't change the settings)

this option never worked really well

It can already happen that you can't fully download a message, because it has been deleted on the server already.

The user sees that there is a failure, and can ask the sender to send the message again

It does make sense to remove only the partial-download logic in the PR here, and then remove the replace_msg_id logic in a later PR. But if there is any kind of effort in it, it's not worth it, esp. as we need pre-messages rather sooner than later for calls etc.

src/imap.rs

src/download.rs

src/mimeparser.rs

src/webxdc/webxdc_tests.rs

src/download.rs

Hocuri

Nice! LGTM once all comments are addressed (i.e. either resolved them, or commented why it's not worth it / doesn't make sense)

BTW, in general, it's good to request a review from some people using the Github UI once you've finished your PR, in order to get speedier reviews

deltachat-rpc-client/tests/test_chatlist_events.py

src/download.rs

src/mimeparser.rs

Hocuri · 2025-11-11T17:50:33Z

src/receive_imf.rs

            info!(context, "Message already partly in DB, replacing.");
            Some(msg.chat_id)
+
+            // TODO: look at this place


It's not that big of a sacrifice:

download-on-demand is just an option (i.e. many people don't have it activated, since they don't change the settings)

this option never worked really well

It can already happen that you can't fully download a message, because it has been deleted on the server already.

The user sees that there is a failure, and can ask the sender to send the message again

It does make sense to remove only the partial-download logic in the PR here, and then remove the replace_msg_id logic in a later PR. But if there is any kind of effort in it, it's not worth it, esp. as we need pre-messages rather sooner than later for calls etc.

src/stock_str.rs

system as it is only ever applied to messages that have a pre-message.

`Config::SimulateReceiveImfError`

and modify it to employ message reordering instead of a partially downloaded instance

`test_download_stub_message`

Simon-Laux · 2025-11-14T22:51:53Z

I don't see how this error is related to the changes in this pr. I'll merge it into the pre-message feature branch anyway now.

error: unresolved link to `Context::set_stock-translation`
   --> src/context.rs:141:11
    |
141 |     /// [`Context::set_stock-translation`] or [`Accounts::set_stock_translation`] calls.
    |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the struct `Context` has no field or associated item named `set_stock-translation`
    |
    = note: `-D rustdoc::broken-intra-doc-links` implied by `-D warnings`
    = help: to override `-D warnings` add `#[allow(rustdoc::broken_intra_doc_links)]`

Hocuri · 2025-11-18T17:15:38Z

I don't see how this error is related to the changes in this pr.

It was fixed by 75eb94e, rebasing on main or merging main will fix the problem

Part of #7367 - Remove partial downloads (remove creation of the stub messages) (#7373) - Remove "Download maximum available until" and remove stock string `DC_STR_DOWNLOAD_AVAILABILITY` (#7369) - Send pre-message on messages with large attachments (#7410) - Pre messages can now get read receipts (#7433) progress / what's to do: - [x] send pre-message - [x] The message's state must be set to MessageState::OutDelivered only after both messages are sent. If a read receipt is received, the message can be OutMdnRcvd or OutPending; let's just do whatever is easiest for now. Take care not to revert from OutMdnReceived to OutDelivered if we first receive a read receipt and then deliver the full message. - this is already the case: - `OutDelivered` is set when a message is sent out and has no remaining send jobs in the smtp table for this message id - so already works since full message and pre message have same msgId in that table. - `OutMdnRcvd` is a "virtual" state (#7367 (comment)), so going back to `OutDelivered` can't happen anymore - [x] delimit `ChatFullMessageId` with `<` and `>` like the other message ids - [x] add tests - [x] test that pre message is sent for attachment larger than X - test that correct headers are present on both messages - also test that Autocrypt-gossip and selfavatar should never go into full-messages - [x] test that no pre message is sent for attachment smaller than X - no "is full message" header should be present - [x] test that pre message is not send for large webxdc update or large text - [x] fix test `receive_imf::receive_imf_tests::test_dont_reverify_by_self_on_outgoing_msg`

Part of #7367 - Remove partial downloads (remove creation of the stub messages) (#7373) - Remove "Download maximum available until" and remove stock string `DC_STR_DOWNLOAD_AVAILABILITY` (#7369) - Send pre-message on messages with large attachments (#7410) - Pre messages can now get read receipts (#7433) progress / what's to do: - [x] send pre-message - [x] The message's state must be set to MessageState::OutDelivered only after both messages are sent. If a read receipt is received, the message can be OutMdnRcvd or OutPending; let's just do whatever is easiest for now. Take care not to revert from OutMdnReceived to OutDelivered if we first receive a read receipt and then deliver the full message. - this is already the case: - `OutDelivered` is set when a message is sent out and has no remaining send jobs in the smtp table for this message id - so already works since full message and pre message have same msgId in that table. - `OutMdnRcvd` is a "virtual" state (#7367 (comment)), so going back to `OutDelivered` can't happen anymore - [x] delimit `ChatFullMessageId` with `<` and `>` like the other message ids - [x] add tests - [x] test that pre message is sent for attachment larger than X - test that correct headers are present on both messages - also test that Autocrypt-gossip and selfavatar should never go into full-messages - [x] test that no pre message is sent for attachment smaller than X - no "is full message" header should be present - [x] test that pre message is not send for large webxdc update or large text - [x] fix test `receive_imf::receive_imf_tests::test_dont_reverify_by_self_on_outgoing_msg` feat: receive pre-messages and adapt download on demand fix python lint errors receive pre-mesages, start with changes to imap loop. refactor: move download code from `scheduler.rs` to `download.rs`, also move `get_msg_id_by_rfc724_mid` to `MsgId::get_by_rfc724_mid` `MAX_FETCH_MSG_SIZE` is no longer unused Parse if it is a pre-message or full-message start with receiving logic get rid of `MsgId::get_by_rfc724_mid` because it was a duplicate of `message::rfc724_mid_exists` docs: add hint to `MimeMessage::from_bytes` stating that it has side-effects. receiving full message send and receive `attachment_size` and set viewtype to text in pre_message metadata as struct in pre-message in header. And fill params that we can already fill from the metadata. Also add a new api to check what viewtype the message will have once downloaded. api: jsonrpc: add `full_message_view_type` to `Message` and `MessageInfo` make PreMsgMetadata.to_header_value not consume self/PreMsgMetadata add api to merge params on download full message: merge new params into old params and remove full-message metadata params move tests to `src/tests/pre_messages.rs` dynamically allocate test attachment bytes fix detection of pre-messages. (it looked for the ChatFullMessageId header in the unencrypted headers before) fix setting dl state to avaiable on pre-messages fix: save pre message with rfc724_mid of full message als disable replacement for full messages add some receiving tests and update test todo for premessage metadata test: process full message before pre-message test receive normal message some serialization tests for PreMsgMetadata remove outdated todo comment test that pre-message contains message text PreMsgMetadata: test_build_from_file_msg and test_build_from_file_msg test: test_receive_pre_message_image Test receiving the full message after receiving an edit after receiving the pre-message test_reaction_on_pre_message test_full_download_after_trashed test_webxdc_update_for_not_downloaded_instance simplify fake webxdc generation in test_webxdc_update_for_not_downloaded_instance test_markseen_pre_msg test_pre_msg_can_start_chat and test_full_msg_can_start_chat test_download_later_keeps_message_order test_chatlist_event_on_full_msg_download fix download not working log splitting into pre-message add pre-message info to text when loading from db. this can be disabled with config key `hide_pre_message_metadata_text` if ui wants to display it in a prettier way. update `download_limit` documentation more logging: log size of pre and post messages rename full message to Post-Message split up the pre-message tests into multiple files dedup test code by extracting code to create test messages into util methods remove post_message_view_type from api, now it is only used internally for tests remove `hide_pre_message_metadata_text` config option, as there currently is no way to get the full message viewtype anymore Update src/download.rs resolve comment use `parse_message_id` instead of removing `<>`parenthesis it manually fix available_post_msgs gets no entries handle forwarding and add a test for it. convert comment to log warning event on unexpected download failure add doc comment to `simple_imap_loop` more logging handle saving pre-message to self messages and test. refactor: Simplify the logic for downloading a bit Immediately fully download all messages that do not have the `Chat-Is-Post-Message` header. This way, we simplify the logic for when which messages are downloaded, there are no differences at all anymore between 'background_fetch' and 'normal fetch'. Also, we prevent message reordering when reveiving a message from a legacy client. Messages larger than 1MB without `Chat-Is-Post-Message` should be rare, so, we do not expect this to really worsen things. Update deltachat-ffi/deltachat.h less .clone() clippy fixes rustfmt restore test_download_on_demand use anyhow remove TODO remove PartialDownloadMsgBody Add DC_STR_CONTACT stock string remove stock string TODOs remove optional todo feat: Don't put text into post-message This way, there will be no duplicate text if a chat partner is using an older DC version. If the chat partner is using a new DC version, then this will have no effect. fix: Make pre-messages db migration backwards-compatible See this comment that described the problem: https://github.com/chatmail/core/pull/7371/files#r2612182387 > The migration removes the `msg_id` column from the `download` table, so that, if you import the backup into an older DC version, you get the error `Failed inbox fetch_idle: Failed to download messages: no such column: msg_id in SELECT msg_id FROM download at offset 7: Error code 1: SQL error or missing database.` > > So, we need: > - either to make sure that things somewhat work if the user downgrades > - or to again increase `backup_version`, and then again coordinate that releases are done on all platforms simultaneously (which I would like to avoid). This PR here implements the first solution: Re-add `download.msg_id`, so that an older core can handle the database scheme. Co-authored-by: Hocuri <hocuri@gmx.de>

- Remove partial downloads (remove creation of the stub messages) (#7373) - Remove "Download maximum available until" and remove stock string `DC_STR_DOWNLOAD_AVAILABILITY` (#7369) - Send pre-message on messages with large attachments (#7410) - Pre messages can now get read receipts (#7433) Co-authored-by: Hocuri <hocuri@gmx.de>

- Remove partial downloads (remove creation of the stub messages) (#7373) - Remove "Download maximum available until" and remove stock string `DC_STR_DOWNLOAD_AVAILABILITY` (#7369) - Send pre-message on messages with large attachments (#7410) - Pre messages can now get read receipts (#7433) Co-authored-by: Hocuri <hocuri@gmx.de> Co-authored-by: iequidoo <dgreshilov@gmail.com>

Simon-Laux force-pushed the simon/remove-partial-download-of-messages branch from 0d46626 to 81ca8c2 Compare October 30, 2025 17:30

Simon-Laux commented Oct 31, 2025

View reviewed changes

src/mimeparser.rs Outdated Show resolved Hide resolved

Simon-Laux mentioned this pull request Oct 31, 2025

[Tracking Issue] Pre-Messages #7367

Closed

7 tasks

Simon-Laux commented Oct 31, 2025

View reviewed changes

Simon-Laux marked this pull request as ready for review October 31, 2025 23:16

Simon-Laux marked this pull request as draft October 31, 2025 23:29

Simon-Laux changed the title ~~remove: partial downloads~~ remove: partial downloads - creation of the stub messages Oct 31, 2025

Simon-Laux marked this pull request as ready for review November 1, 2025 12:41

iequidoo reviewed Nov 3, 2025

View reviewed changes

src/imap.rs Outdated Show resolved Hide resolved

Simon-Laux force-pushed the pre-messages branch from 6fe6723 to 8140ebc Compare November 6, 2025 13:13

Simon-Laux force-pushed the simon/remove-partial-download-of-messages branch from 6c84e01 to 938c890 Compare November 6, 2025 13:22

Simon-Laux assigned Simon-Laux and unassigned Simon-Laux Nov 6, 2025

Simon-Laux mentioned this pull request Nov 8, 2025

feat: pre-messages / next version of download on demand #7371

Merged

9 tasks

Simon-Laux changed the title ~~remove: partial downloads - creation of the stub messages~~ remove: partial downloads (remove creation of the stub messages) Nov 8, 2025

iequidoo mentioned this pull request Nov 10, 2025

Don't prefetch Chat-Version header and don't copy it into outer part #7405

Draft

iequidoo reviewed Nov 10, 2025

View reviewed changes

src/download.rs Show resolved Hide resolved

src/mimeparser.rs Outdated Show resolved Hide resolved

src/webxdc/webxdc_tests.rs Show resolved Hide resolved

src/download.rs Show resolved Hide resolved

src/download.rs Show resolved Hide resolved

Hocuri reviewed Nov 11, 2025

View reviewed changes

Simon-Laux mentioned this pull request Nov 12, 2025

feat: receive pre-messages and adapt download on demand #7431

Merged

26 tasks

Simon-Laux force-pushed the simon/remove-partial-download-of-messages branch from 506ecc8 to e564a9b Compare November 12, 2025 19:36

Simon-Laux and others added 9 commits November 12, 2025 20:37

remove: partial downloads

cca4174

remove partial download tests and fix issues in other tests

fdcb222

remove remaining partial messages tests

7f3eabc

remove download limit validation, it does not really matter in the new

2e2bc12

system as it is only ever applied to messages that have a pre-message.

fix clippy

902a0d7

ignore that partial_download_msg_body is dead code

70d7e04

remove test for download limit validation

2284d36

remove test_download_on_demand

53163ab

remove test test_download_limit_chat_assignment

cc9fe5d

Simon-Laux and others added 10 commits November 12, 2025 20:37

fix python lint errors

014d80b

remove cffi python testtest_webxdc_download_on_demand

a044436

fix python lint

28cca8b

fix rebase issue

560e86b

fix another rebase issue

5a2ca7a

fix lint of python test

10c2016

reintroduce Config::FailOnReceivingFullMsg but renamed it to

37b027a

`Config::SimulateReceiveImfError`

allow exclusion of simulate_receive_imf_error from get_info

afa46c8

adapt comment

6d11e08

remove StockMessage::PartialDownloadMsgBody

05a1967

Simon-Laux force-pushed the simon/remove-partial-download-of-messages branch from e564a9b to 05a1967 Compare November 12, 2025 19:38

Simon-Laux added 5 commits November 14, 2025 18:23

fix clippy issue

b70a41f

restore test test_downloadstate_values

971286b

restore test test_webxdc_update_for_not_downloaded_instance and rename

af00375

and modify it to employ message reordering instead of a partially downloaded instance

bring back receiving half of test_partial_receive_imf and rename to

ab1fdfa

`test_download_stub_message`

remove test_webxdc_update_for_not_yet_received_instance

f7d8812

Simon-Laux merged commit fa24dee into pre-messages Nov 14, 2025
27 of 28 checks passed

Simon-Laux deleted the simon/remove-partial-download-of-messages branch November 14, 2025 22:52

iequidoo mentioned this pull request Nov 16, 2025

Support reordering of webxdc instances and updates #7457

Open

Uh oh!

remove: partial downloads (remove creation of the stub messages) #7373

remove: partial downloads (remove creation of the stub messages) #7373

Uh oh!

Conversation

Simon-Laux commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

removed partial download tests:

Uh oh!

Simon-Laux commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Simon-Laux Oct 31, 2025

Choose a reason for hiding this comment

Uh oh!

Simon-Laux Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Hocuri Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Hocuri left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Hocuri Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Simon-Laux commented Nov 14, 2025

Uh oh!

Uh oh!

Hocuri commented Nov 18, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Simon-Laux commented Oct 30, 2025 •

edited

Loading

Simon-Laux commented Oct 30, 2025 •

edited

Loading

Simon-Laux Oct 31, 2025 •

edited

Loading

Hocuri left a comment •

edited

Loading