Importing messages silently skips some based on sub_id

[wittmane]

I exported my messages to transfer them to another device, and the app specified how many SMS and MMS messages were exported, and on the other device it said that same number of messages were imported, but when I went to my SMS app, I noticed it was missing messages. I then tried exporting the messages from the new device, and it said it exported less messages than before.

I did some digging in the export files to compare the difference between messages that got imported and those that failed, and I found that if sub_id was set to 2, they didn't import. I found a mix of -1, 1, and 2 for the values. I'm not sure what that means, but the import skipped 2 for some reason.

I created a sample file to test:

{"_id":"3","thread_id":"1","address":"1235551234","date":"1672531320000","date_sent":"0","read":"1","status":"-1","type":"2","body":"Test message with sub_id 2","failure_cause":"-1","locked":"0","sub_id":"2","stack_type":"0","error_code":"0","creator":"com.google.android.apps.messaging","seen":"1","pri":"0","rcs_extra":"0","rcs_timestamp_delivered":"0","rcs_timestamp_displayed":"0","rcs_message_type":"0","__display_name":"John Doe"}
{"_id":"2","thread_id":"1","address":"1235551234","date":"1672531260000","date_sent":"0","read":"1","status":"-1","type":"2","body":"Test message with sub_id 1","failure_cause":"-1","locked":"0","sub_id":"1","stack_type":"0","error_code":"0","creator":"com.google.android.apps.messaging","seen":"1","pri":"0","rcs_extra":"0","rcs_timestamp_delivered":"0","rcs_timestamp_displayed":"0","rcs_message_type":"0","__display_name":"John Doe"}
{"_id":"1","thread_id":"1","address":"1235551234","date":"1672531200000","date_sent":"0","read":"1","status":"-1","type":"2","body":"Test message with sub_id -1","failure_cause":"-1","locked":"0","sub_id":"-1","stack_type":"0","error_code":"0","creator":"com.google.android.apps.messaging","seen":"1","pri":"0","rcs_extra":"0","rcs_timestamp_delivered":"0","rcs_timestamp_displayed":"0","rcs_message_type":"0","__display_name":"John Doe"}

As I expected, when I imported, it reported 3 SMS imported, but my SMS app only showed the 2 messages with sub_id set to -1 and 1, but skipped the one set to 2.

I did a simple find and replace on my original export to change all "sub_id":"2" to "sub_id":"1", and I was able to do the import successfully with that.

[tmo1]

Thanks for reporting this - I have confirmed the problem. I see that others have reported similar problems and solutions with the SMS Backup and Restore app (which was an inspiration for SMS Import / Export) - see this thread, and this message and this one in particular.

This is apparently a new problem with Android 14 / API level 34 - I reproduced the problem with the file you provided on an API level 34 AVD, but all three messages imported properly on an API level 31 AVD.

I'm not sure what to do with this - at this point, it seems to be an Android problem rather than an app problem, but it may be worth implementing some sort of workaround. I'll have to keep an eye on that thread and the internet in general to see how this shakes out.

[dshepsis]

I experienced a very similar issue. I tried exporting a few years worth of SMS/MMS messages from a Pixel 3a running stock Android 12 to a Pixel 7 running Grapheneos Android 14. I used version 2.2.0 of the app (installed via the APK provided in the GitHub Releases section of this repo). I exported several thousuand SMS and MMSs from the Pixel 3a, copied the ZIP file over to the Pixel 7, and ran the Import Messages function, which shows that the same number of SMS and MMS were imported. I checked the data in the zip file and it appears to all be present. However, when I then opened the messaging app, only 3 messages from one sender were successfully imported. All the other messages appear to be missing.

I tried clearing the app storage and cache, re-exporting and importing, and using another messaging app (initially using the default Grapheneos Messaging app, then tried QKSMS) and all produced the same results, with only the same 3 messages showing up.

The only difference in my case is that the 3 messages which were imported all had "sub_id":"-1", whereas every other message had "sub_id":"1". There were none with a value of 2 or anything else.

To fix this, I followed these steps:

1. Extract the ZIP file produced by the app on my desktop PC
2. Open the messages.ndjson file
3. Use Notepad to Replace All "sub_id":"1" with "sub_id":"-1"
4. Make a new ZIP file containing the ndjson file and the data folder.
5. Transfer the modified ZIP to the Pixel 7.
6. Use the "Import Messages" option in the SMS-IE app on the modified ZIP.

This worked, and all my SMS and MMS messages were now visible in both the default Messaging app and QKSMS.

I'm not sure what the intended use and meaning of sub_id is, so I can't suggest a universal solution. For wittmane, a value of 1 worked, and -1, 1, and 2 were all present. For me, only -1 worked, and there were no messages with 2. Maybe it would work for everyone to set the value to -1 for all messages?

[tmo1]

Maybe it would work for everyone to set the value to -1 for all messages?

I think I know what's going on: the sub_id field contains a "Subscription ID." The official documentation explains:

The Subscription ID provides an index value (starting at 1) for uniquely identifying installed SIMs (including physical and electronic) used on the device. Through this ID, your app can associate its functionality with various subscription information for a given SIM. This value is stable for a given SIM unless the device is factory reset. However, there may be cases where the same SIM has a different Subscription ID on different devices or different SIMs have the same ID on different devices.

It seems that up until Android 14, Android would happily insert messages regardless of their sub_id values, but now it rejects those with improper (from the perspective of the target device) values. The reason a value of -1 works is that negative values indicate that "the sub id cannot be determined", so the messages are not rejected.

One solution would therefore be to just automatically (perhaps controlled by a settings toggle) set all sub_id values to -1 upon import; a more subtle but complicated solution might be to check for improper values and set just those to -1. I'm still trying to figure out, though, exactly what values are considered improper.

[tmo1]

I've pushed a commit that sets all sub_ids to -1 upon import (the old behavior is still available via a settings toggle). Please test and report the results here.

[zanechua]

So my guess is that the new api behavior only imports the messages for the sub id that exist on the phone?

For example:

If you had two sims turned on, with sub id 1 and 2 respectively, I suspect that Android would import messages with sub id 1 and 2 just fine.

But if the device only contained 1 sim, then it would only import the messages corresponding with that Sim's sub id.

Haven't tested this out

[sshaikh]

Another data point as this seems the most comprehensive coverage of this issue (I use SMS Backup and Restore, but came across this app while investigating the issue). Unlike most reports, I wasn't restoring or migrating, but only noticed the behaviour while backing up post Android 14, so I think it's worth documenting that it's not just about trying to restore new messages:

1. I backup SMSs daily
2. On the morning of the 4th October I had a backup file with 300 messages
3. I upgraded to Android 14 in the afternoon of 4th October
4. On the morning of the 5th October I had a backup file with 275 messages
5. I realised the messages were being dropped from backup on the 31st October (albeit since the 5th)
6. I noticed that the messages were also actually missing from Google Messages, and so it also saw 275 messages. Strangely, Chomp showed the missing messages in the list view, but not when opening the conversation (I put this down to a strange internal caching issue). The point being that SMS backup was consistent with what I could see in Messages.
7. Doing a diff of the 4th and 5th backup files, I saw that the missing 25 messages all had a strange sub_id: 24 were contiguous (time wise) with a sub id of 4, 1 was at a different time with a sub_id of 7. I assume I used a different phone or something during those times.
8. I restored the missing messages by creating a restore file with the 25 messages, correcting their sub_id to 1. This brought my sms total back up to 300.

Open questions:

1. Were the 25 messages deleted by the Android upgrade?
2. Or were they deleted by a misbehaving app post upgrade?
3. Or were they actually not deleted at all, but hidden?

[tmo1]

Another data point as this seems the most comprehensive coverage of this issue (I use SMS Backup and Restore, but came across this app while investigating the issue).

This is a great contribution toward figuring out this issue - thank you!

Strangely, Chomp showed the missing messages in the list view, but not when opening the conversation (I put this down to a strange internal caching issue).

I'll bet it's indeed a caching issue - see here (paragraph beginning "Note").

Open questions: ...

I suspect #1 or #3 - this would explain something that's been bothering me about this whole issue: if Android 14 has simply begun to refuse to insert messages with "improper" Subscription IDs into the message databases, why isn't it reporting this failure in response to the insert call? Based on your analysis, it seems that it's not actually the import that's failing; rather, Android is subsequently deleting or hiding messages with Subscription IDs that it doesn't like.

[sshaikh]

Yes, I was hoping another would come to the same conclusion - it may also explain the Chomp behaviour, as the counts may be reflecting hidden duplicates with bad sub_ids

[sshaikh]

https://xdaforums.com/t/solved-sms-backup-restoring-to-wrong-sim.4580147/

Might be worth digging into this db?

[tmo1]

it may also explain the Chomp behaviour, as the counts may be reflecting hidden duplicates with bad sub_ids

You can check whether the problem is due to caching or something else by clearing the cache, as per the link I posted above, and seeing whether the discrepancy remains.

Might be worth digging into this db?

See also here.

[sshaikh]

It actually never occurred to me to have messages associated to a SIM card (vs phone number which makes a bit more sense to me).

If they are being suppressed due to a missing entry in the sub db then an alternative solution might be to add that entry in the db.

Edit: I just cleared the data from Chomp and the counts are still incorrect (but make sense if the bad sub_id messages are still being counted)

[tmo1]

If they are being suppressed due to a missing entry in the sub db then an alternative solution might be to add that entry in the db.

I'm not sure what that would mean. Android has the SubscriptionManager API, but that seems to be primarily for getting information about subscriptions. I'm not sure how one would add arbitrary SubscriptionID entries to Android, and whether that would be a good idea.

[wittmane]

Sorry about the delay, but I finally got around to testing, and the update fixed my issue. I see that there has been some discussion after the fix was pushed, so I'm not sure if you still want this open for tracking something with that, but as far as my original issue goes, I'm content to close the issue.

[jayb-g]

The issue and the solution(the app converting all sub_id to -1) also makes SMS apps to show imported messages timestamps to fall on the date of import rather the date of sent/received in the list view. Upon opening the message, it shows correct date time.

[tmo1]

The issue and the solution(the app converting all sub_id to -1) also makes SMS apps to show imported messages timestamps to fall on the date of import rather the date of sent/received in the list view. Upon opening the message, it shows correct date time.

Thanks for reporting this; I'm looking into it.

[Neob91]

I'm not sure if I should create a separate issue, because it's not strictly related to sub IDs, but I found that messages having spaces in the address field (e.g. +48 123 456 789) will also not be imported. I hope this helps save others some time.

[tmo1]

I found that messages having spaces in the address field (e.g. +48 123 456 789) will also not be imported

Thank you for reporting this - I'll take a look.

[tmo1]

I found that messages having spaces in the address field (e.g. +48 123 456 789) will also not be imported.

Thanks for reporting this problem - further discussion of it is occuring in issue #159, beginning here.

[julianfoad]

I would like to suggest that the meaning of the new setting "Include Subscription IDs in import" is too obscure to be a generally visible setting without further explanation. I came across this when setting up for a restore for the first time, and it took me quite some digging to find this issue where the problem and work-around is discussed, and even then I'm not 100% sure whether it will work in my current case.

If we can come up with an attached explanation, something simple but complete enough, then that explanation should be included. Otherwise, I should think the option should be in the "Debugging options" section.

A really good solution might involve scanning the sub-IDs present in a requested import, comparing them with the sub-IDs reported by the phone, and taking different actions or warn/ask the user what to do based on whether that looks like a good enough match. (Details to be fleshed out.)

[tmo1]

I would like to suggest that the meaning of the new setting "Include Subscription IDs in import" is too obscure to be a generally visible setting without further explanation. I came across this when setting up for a restore for the first time, and it took me quite some digging to find this issue where the problem and work-around is discussed, and even then I'm not 100% sure whether it will work in my current case.

The "Include Subscription IDs in import" setting is clearly documented in the README.

A really good solution might involve scanning the sub-IDs present in a requested import, comparing them with the sub-IDs reported by the phone, and taking different actions or warn/ask the user what to do based on whether that looks like a good enough match. (Details to be fleshed out.)

If someone wants to flesh out the details and / or write some code, I'll be happy to take a look :)