exposure of partial writes with encryption (and multiprocess)

[finnschiermer]

When encryption is enabled, we write sectors of the file in a non-atomic fashion. In case the writing process is aborted (app process terminated) in the midst of such a write, the sector will only be partially written, leading to a corrupted file.

It is unclear if this bug has been observed IRL, but it is clearly possible.

This problem is even more important if/when we want to support multi-process encryption, because it allows another process to observe partial writes (by reading at the wrong moment), which will be seen as file corruption, even when the writes later complete and the file is not corrupt.

This bug has been present since our first implementation of encryption.

[finnschiermer]

Initial possible solution:

wrt. incomplete sector write due to process termination:

Before writing a sector, write its payload and metadata to a separate file (the "pending-write-file") and sync it to disk. When a realm is first opened, a check is made and if a valid pending write exists, it is performed and the pending-write-file is marked invalid.
After syncing the pending-write-file, commence writing the sector in the main realm file as usual. When done, mark the pending-write-file invalid again.

wrt. other processes seeing torn writes (in a multiprocess scenario):

Protect against this by using file locking on the realm file whenever a sector is written (exclusive lock) or read (shared lock)

[finnschiermer]

Update: We've built a special version of realm-core which produces a copy of the realm file at different (pseudo-random) points during the writing process. This generates many thousands files each corresponding to a potential crash-point. These files are subsequently decrypted and checked for consistency. So far no smoking gun, but a) the workload may not be right and b) the consistency checks not strict enough. Yet.

[palaniraja]

@finnschiermer can we have the link to the special version of realm-core? and is it part of realm-swift (v10.23.0) released 4 days ago?

[finnschiermer]

@palaniraja sorry for the late reply: We don't want to do an actual release containing something like this. I'll push a branch with it tomorrow.

We're proceeding to fix the issue described here (for single process scenarios), however my current assessment is that this is unfortunately not the root cause of any of the reported errors.

[driq]

The report at realm/realm-swift#7629 suggests that this might be a regression introduced in realm-core v11.7.0 or v11.8.0

[finnschiermer]

The torn write problem (this issue) is much older, so there must be some other root cause for the recent reports of corruption with encryption enabled. We are investigating.

[sync-by-unito]

➤ Finn Andersen commented:

The issue described here is now considered theoretical, should not occur on any existing device we know of. The problem will be fixed later as part of building out support for multprocess use of encryption, so not closing this issue yet.

The issues linked/marked as depending on this issue, should instead be marked as depending on and is fixed by https://github.com/realm/realm-core/pull/5383.

[sync-by-unito]

➤ finnschiermer commented:

This is now handled by the work on multiprocess encryption.