JSON format documentation

[hydrargyrum]

I'm the author of Epistolaire, an android app for exporting sms/mms to JSON. Unfortunately, it cannot import data yet.

To have better compatibility between our apps, I'm interested in what JSON format you use in your app. Epistolaire's format is described at https://github.com/hydrargyrum/epistolaire#json-format= and https://github.com/hydrargyrum/epistolaire/blob/master/backup.schema.json

Do you have a JSON schema description or a brief description? When importing, do you skip extra fields or do you reject the whole JSON?

This is related to another app which uses the same format as Epistolaire: hydrargyrum/breakthesilence#1

[tmo1]

I'm the author of Epistolaire, an android app for exporting sms/mms to JSON.

Welcome! Our apps are indeed similar, both in what they do, as well as in the philosophy behind them (data freedom and privacy, avoidance of the Play Store).

Unfortunately, it cannot import data yet.

Had Epistolaire had import capability at the time I wrote SMS Import / Export, I might never have written it ;)

Do you have a JSON schema description or a brief description?

No schema - description here. One key difference between my JSON and yours is that I don't use conversation structures - perhaps I should. My thinking was that (IIUC) the notion of a "conversation" is not inherent to the messages themselves, but an organizational scheme layered on top of them by Android, but I can see that it might be helpful to preserve the Android conversation structure in the JSON.

I've been wanting to improve my JSON structure for a while, but I want to be quite sure before I take the plunge, since it's going to break compatibliity between versions (and / or require writing a migration tool), require a fair bit of careful recoding, and will likely introduce new bugs ;)

One thing I probably want to add is backup metadata (like your errors tag).

When importing, do you skip extra fields or do you reject the whole JSON?

The former.

This is related to another app which uses the same format as Epistolaire: hydrargyrum/breakthesilence#1

Ah, I see we also share qualms about Moxie's attitude and some of Signal's design decisions ;)

[hydrargyrum]

Had Epistolaire had import capability at the time I wrote SMS Import / Export, I might never have written it ;)

Hehe, I'm a bit sorry I never took the time to implement it yet, but I'm happy to see other apps, more choice is better! And I should really give a try to SMS Import / Export app as it seems to fulfill several feature requests users made on Epistolaire.

No schema - description here. One key difference between my JSON and yours is that I don't use conversation structures - perhaps I should. My thinking was that (IIUC) the notion of a "conversation" is not inherent to the messages themselves, but an organizational scheme layered on top of them by Android, but I can see that it might be helpful to preserve the Android conversation structure in the JSON.

I totally understand. My reasoning behind introducing conversations myself was that it would be easier to exploit the JSON for display for example, I hadn't importing in mind at that time (else, the decision would have probably been different).

I've been wanting to improve my JSON structure for a while, but I want to be quite sure before I take the plunge, since it's going to break compatibliity between versions (and / or require writing a migration tool), require a fair bit of careful recoding, and will likely introduce new bugs ;)

Since SMS Import / Export app can import messages, I think I'll redirect some of my users to it (until I finally implement some import myself). I may write a conversion script for Epistolaire format to SMS Import / Export format, I did a few converters already.

One thing I probably want to add is backup metadata (like your errors tag).

Dump timestamp etc.? About errors, it helps but not always, for example some users have encountered invalid JSON files with Epistolaire, and of course they can't share the data so it's very hard to reproduce :(

When importing, do you skip extra fields or do you reject the whole JSON?

The former.

Great! That will avoid errors about futile conversion details.

Ah, I see we also share qualms about Moxie's attitude and some of Signal's design decisions ;)

Now I'm laughing behind my computer, it seems we've had many common troubles :)

[tmo1]

Great! That will avoid errors about futile conversion details.

My initial code just naively tried to insert all fields (except the ones we add on export). This worked fine in testing, since I was exporting and importing from AVDs running the same (or similar) versions of Android, but in the real world, people started seeing crashes (#11, #12, #14), so I realized that I had to do something about fields added in later Android versions. My solution was to just drop those fields and hope that the import still works - so far there have been no reports that it doesn't ;)

Now I'm laughing behind my computer, it seems we've had many common troubles :)

I'm very unhappy about the phone number requirement, although they're apparently considering / working on changing that, and rather annoyed by the refusal to provide an unencrypted and documented backup / export of the message store, necessitating third party applications (which still only solves the encryption issue, not the lack of documentation).

I respect Moxie, both for his technical skills and for his good intentions, but I dislike his attitude of "we're going to do this the way we think is best, since we're the experts, regardless of user preferences" as well as the lack of interest or even hostility toward open standards and interoperability.