Bad Realm file header (#3)

[beeender]

One of realm-java user gets crashes with below backtrace

Fatal Exception: io.realm.exceptions.RealmFileException: Unable to open a realm at path '/data/data/com.xxx.android/files/xxxxx.realm': Bad Realm file header (#3). in /Users/cm/Realm/realm-java/realm/realm-library/src/main/cpp/io_realm_internal_SharedRealm.cpp line 81
       at io.realm.internal.SharedRealm.nativeGetSharedRealm(SharedRealm.java)
       at io.realm.internal.SharedRealm.getInstance(SharedRealm.java:178)
       at io.realm.internal.SharedRealm.getInstance(SharedRealm.java:155)
       at io.realm.RealmCache.createRealmOrGetFromCache(RealmCache.java:124)
       at io.realm.Realm.getInstance(Realm.java:228)

It happens to Android 6.0,4.4.4,5.0.2.
It cannot be reproducible from user side right now.
around 0.05% users met this issue.

[finnschiermer]

This refers to a situation where the top-ref points to a top-array lying beyond the logical end of the file.

[finnschiermer]

Could be the same root cause as realm/realm-java#3264 / tracked as Core issue #2230

[finnschiermer]

@beeender Any chance of a) different paths are being used by different threads, but for the same database (aliasing) ? or b) this involves sharing among multiple processes?

[ironage]

I checked a few of the problematic devices reporting crashes from realm/realm-java#3264 and they are using 32-bit CPUs. This information, in addition to the note by @finnschiermer that

the top-ref points to a top-array lying beyond the logical end of the file.

leads me to believe that the problem is the same one that @danielpovlsen is tracking down where the allocator somehow generates memory addresses which are too high on 32 bit devices (Daniel can reproduce this on the iPhone 5, 32-bit iOS simulator but I cannot). @danielpovlsen do you have an issue with more information on this?

[finnschiermer]

This could point to a problem with managing the shared memory mappings, then.

[mihirjoshi21]

This issue is linked to #2984. Here is the crash log http://crashes.to/s/706cc3e62fc.

[mihirjoshi21]

This is the only bug left even after updating to 2.2.1. Almost 16-17 crashes every day. Please fix this.

[finnschiermer]

We need a repro to make progress.

[bmunkholm]

@finnschiermer could we in the mean time add more logging info to get further clues?

[ironage]

@mihirjoshi21 are you using encrypted Realm files?

[mihirjoshi21]

@ironage No realm file is not encrypted.

[mihirjoshi21]

Any update on this?

[teotwaki]

@mihirjoshi21 Do you have a reproduction case for this?

[mihirjoshi21]

@teotwaki Only if I could reproduce it but I can't. There is no pattern in the crash and not specific to any android version or device types. This is the crash log http://crashes.to/s/706cc3e62fc.

[beeender]

realm/realm-java#5063

We have more and more user reporting this issue. @finnschiermer @ironage I got some information from user which can be found in realm/realm-java#5063 But the information we got doesn't seem to be helpful to identify the issue. Please, if you have something in mind, just ask @AlexTip who reported it to get more information.

[AlexTip]

Hi there. I'll provide all the information I can on demand.
By the way if it's really vital I can take a chance to rollback Realm version to 3.1.4 and push it to production in order to see if number of crashes will fall down.

[Zhuinden]

Realm-Java 3.2.2 (2017-05-19):

Internal

Upgraded to Realm Sync 1.8.5.
Upgraded to Realm Core 2.8.0.

---

Realm-Java 3.1.3 (2017-04-20)

Internal

Upgraded to Realm Sync 1.6.0.
Upgraded to Realm Core 2.6.1.

I guess the question is whether bug occurs from core or object store.

[beeender]

@AlexTip
realm-java 3.5.0 is using realm-core 2.8.6. 3.1.4 is using realm core 2.6.1.
From what i can tell, it is fine to roll back from realm-java 3.5.0 to 3.1.4.

@finnschiermer @ironage please correct me if i am wrong.

[finnschiermer]

I'm sorry, but we need to be able to reproduce this before we can make further progress.

[bmunkholm]

@paulVulog @ptsiogas It's worth noting that this is not a "specific bug", but an indication that your realm got corrupted. That could be for entirely different reasons. So it's super valuable to get as much info as you can provide about your specific circumstances.
How frequent are you seing this (also in % of users)? In production? which devices? how much disk is left on the device? Anything else that can characterize the occurrences?
Thanks!

[harryoik]

@bmunkholm In production it occurs at 0.4% of the users. The 91% occurs in devices with Android 4 and 86% are Samsung. They all seem to have enough free space in their devices but most of them have less than 20% of the RAM available.
The issues have increased after migrating to an encrypted realm and also they are usually repeatable.

[finnschiermer]

@harryoik In the scenario you describe above, where less than 20% of RAM is available, what is the size of your realm file relative to the RAM size on those devices? Also, what is the absolute size of your realm file?

[bmunkholm]

@harryoik Thanks for those details! Once the realm file is corrupted, it will always repeat the same error when the app is restarted. The realm file has to be replaced in order to recover. So when you say it's repeatable, do you mean the error is repeated when the app is restarted? If you have a case where you can reproduce the situation where the app is fine, but doing something corrupts the file so that when the app is restarted you get the error - then we are super interested to learn how we can reproduce that.

[harryoik]

@finnschiermer I don't know the exact size of the realm file in those devices but it will be about 4-5MBs so it's pretty small.
@bmunkholm I haven't managed to reproduce this crash. By repeatable I mean that it's repeated every time the app restarts. How can I replace realm's file in order the issue not to be repeated;

[paulVulog]

Do you have a work around at application start to:

• first detect if the realm file is corrupted
• then reset the file in order to continue the application launch

[bmunkholm]

Unfortunately not. But we just discussed to create a proposal for that: See #3031

[mihirjoshi21]

       at io.realm.internal.OsSharedRealm.nativeGetSharedRealm(OsSharedRealm.java)
       at io.realm.internal.OsSharedRealm.(OsSharedRealm.java:171)
       at io.realm.internal.OsSharedRealm.getInstance(OsSharedRealm.java:241)
       at io.realm.internal.OsSharedRealm.getInstance(OsSharedRealm.java:231)
       at io.realm.RealmCache.doCreateRealmOrGetFromCache(RealmCache.java:319)
       at io.realm.RealmCache.createRealmOrGetFromCache(RealmCache.java:282)
       at io.realm.Realm.getDefaultInstance(Realm.java:343)
       at com.conem.app.pocketthesaurus.display.FragmentActivityMain.onCreate(FragmentActivityMain.java:112)
       at android.app.Activity.performCreate(Activity.java:6074)

So this bug is consistent since beta days of realm. Now the app has over 400k users and it occurs for less than 1 percent of the people so it's pretty rare. There is no pattern like certain devices, low memory or android version. In almost 2 years I have never been able to reproduce it. Some bugs you just live up with.

[bmunkholm]

Which version do you use?

[mihirjoshi21]

@bmunkholm 5.0.0

[jpage4500]

The same crash has happened 28k times (425 different users) for the app I'm working on as well. We're now on 5.0.0 but doesn't seem to make a difference. Also seems to happen on many devices and OS's.

The only thing I did notice in Fabric's dashboard view is that 89% of the time this happened the app was running in the background.

Caused by io.realm.exceptions.RealmFileException: Unable to open a realm at path '/data/data/com.quirky.android.wink.wink/files/default.realm': Bad Realm file header (#3). (Bad Realm file header (#3)) (/data/data/com.quirky.android.wink.wink/files/default.realm) in /Users/cm/Realm/realm-java/realm/realm-library/src/main/cpp/io_realm_internal_OsSharedRealm.cpp line 101
       at io.realm.internal.OsSharedRealm.nativeGetSharedRealm(OsSharedRealm.java)
       at io.realm.internal.OsSharedRealm.(OsSharedRealm.java:171)
       at io.realm.internal.OsSharedRealm.getInstance(OsSharedRealm.java:241)
       at io.realm.internal.OsSharedRealm.getInstance(OsSharedRealm.java:231)
       at io.realm.RealmCache.doCreateRealmOrGetFromCache(RealmCache.java:319)
       at io.realm.RealmCache.createRealmOrGetFromCache(RealmCache.java:282)
       at io.realm.Realm.getDefaultInstance(Realm.java:343)

[bmunkholm]

@jpageo4500 Wauu that's a lot... Sorry to hear that. Once you have a corrupt file it will keep crashing no matter how many times you restart the app. One crude way to handle this is to catch the exception and delete the Realm and restart the app.
But it's obviously most interesting to figure out how it get's corrupted in the first place. If you could possibly get your hands on any of those realms that would be gold.

In the meantime, we should consider an update with more information about how the file is corrupted to see if there is anything we can learn from that. @finnschiermer @jedelbo other ideas?

[paulVulog]

Still having the issue here too. It has a pretty huge impact on user experience since as explained, as soon as the realm file is corrupted, you get a crash every-time you call : Realm.getDefaultInstance()

The only hack for now is :

• catch the crash
• remove the realm file
• create a new default configuration

Here is my sample code at application initialization time :

		//init Realm
		Realm.init(appContext);
		RealmConfiguration realmConfiguration = new RealmConfiguration.Builder().build();
		Realm.setDefaultConfiguration(realmConfiguration);

		//Force crash if any
		try {
			Realm realm = Realm.getDefaultInstance();
		} catch(RealmFileException e) {
			Realm.deleteRealm(realmConfiguration);
			RealmConfiguration newRealmConfiguration = new RealmConfiguration.Builder().build();
			Realm.setDefaultConfiguration(newRealmConfiguration);
		}

Less than 1% of crash is still a lot ! A fix needs to be created.

• Looks like it is a corrupted file header. Can't we just fix the header with a kind of backup in case it got corrupted ? Preventing the full file to be lost ?
• Is this crash reported on custom realm configuration ? (in regards of default configuration)

On my side I have no other specific information regarding this crash.
It just happens now while I was developing on my Nexus 5x. (android 8.1, not rooted).
Nothing noticed in my crash repport. (all OS, all devices ....)

[jpage4500]

thanks @paulVulog - I was wondering if I could recover from this Exception and the example you're using above seems like a good idea. Certainly better than letting the app crash (more so if the user will never be able to restart it again w/out clearing the app cache/data)

I'm still interested in finding out what might have caused this corrupted file in the first place and will keep searching. I'll also log a non-fatal Exception to Fabric to help narrow the search down. Is there anything useful I can add to my logging at the time this happens? I was a little interested to see that Fabric reported most of the crashes took place while the app was in the background.. I wonder if Android is killing some long-running background process that's taking too long which is creating this situation in the first place

[bmunkholm]

Thanks @paulVulog for sharing this. The header contains a cookie to detect that this is indeed a realm file. It's never supposed to be overwritten, and trying to repair the file is not the right solution as a lot of other stuff could be wrong.
@jpage4500 The most valuable at this time is if it is possible to upload the realm file when your crash tool detects the crash. (assuming that's doable within your privacy policy).

[paulVulog]

Sorry, I forgot to say that @jpage4500 :
In my case, looks like it is not happening while the app is in background. (fabrics : app in background < 1%)
But it means nothing, because we need to check when the file get corrupted. It's not the same as when you get the crash. (Realm.getDefaultInstance() call)

In my case, I have no crash reported until the application initialization (i.e : application restart).
It means the realm file is probably corrupted only when the application is terminating (system or user termination).

Thanks for the explanation @bmunkholm . If the header is never overwritten, is it possible that the "header reader" return a false error while checking if it is a realm file ?

[bmunkholm]

If the header is never overwritten, is it possible that the "header reader" return a false error while checking if it is a realm file ?

That seems highly unlikely. But I suggest we log what in the header is considered wrong, and possibly the whole header so we can see if it's been completely overwritten in some way. (Tracking here: #3075)

[jedelbo]

If you get a Bad Realm file header (#3) exception, the magic characters are ok. Then it is just the top_ref pointing beyond end of file.

[jedelbo]

These kinds of errors are now tracked in #3133. Closed as a duplicate.