Error when deleting RealmObject: Key not found

[GaelCO]

Goal

I have some weird errors on deleting object, but I don't understand why it's happened.
Unfortunately, I don't encountered this issue myself, but according to crashlytics data, lot of users have this bug.

Actual Results

Stack Trace by firebase Crashlytics :

Non-fatal Exception: io.realm.exceptions.RealmError: Unrecoverable error. Key not found in /Users/cm/Realm/realm-java-release/realm/realm-library/src/main/cpp/io_realm_internal_Table.cpp line 280
       at io.realm.internal.Table.nativeMoveLastOver(Table.java)
       at io.realm.internal.Table.moveLastOver(Table.java:315)
       at io.realm.RealmObject.deleteFromRealm(RealmObject.java:117)
       at io.realm.RealmObject.deleteFromRealm(RealmObject.java:89)
       at com.samsic.contrattravail.data.factory.DocumentFactory$Companion.removeDocument(DocumentFactory.java:599)
       at com.samsic.contrattravail.data.factory.DocumentFactory$Companion.deleteDocumentsById(DocumentFactory.java:592)
       at com.samsic.contrattravail.data.factory.DocumentFactory$Companion.deleteDocuments$app_prodRelease(DocumentFactory.java:585)
       at com.samsic.contrattravail.data.factory.DocumentFactory$Companion.deleteCarteVitale(DocumentFactory.java:414)
       at com.samsic.contrattravail.data.factory.SalarieFactory$Companion.deleteSalarie(SalarieFactory.java:339)
       at com.samsic.contrattravail.data.factory.SalarieFactory$Companion.deleteSalarieIfUnused(SalarieFactory.java:438)
       at com.samsic.contrattravail.data.factory.SalarieFactory$Companion.removeUnusedTabletSalarie(SalarieFactory.java:387)
       at com.samsic.contrattravail.data.helper.SynchroReferentielHelper$Companion$cleanDataOnLaunch$1.write(SynchroReferentielHelper.java:137)
       at com.androidcore.realm.DatabaseWriteHelper$getWriterCompletable$1$1.execute(DatabaseWriteHelper.java:113)
       at io.realm.Realm.executeTransaction(Realm.java:1489)
       at com.androidcore.realm.DatabaseWriteHelper$getWriterCompletable$1.subscribe(DatabaseWriteHelper.java:112)
       at io.reactivex.internal.operators.completable.CompletableCreate.subscribeActual(CompletableCreate.java:39)
       at io.reactivex.Completable.subscribe(Completable.java:2309)
       at io.reactivex.internal.operators.single.SingleFlatMapCompletable$FlatMapCompletableObserver.onSuccess(SingleFlatMapCompletable.java:91)
       at io.reactivex.internal.operators.single.SingleSubscribeOn$SubscribeOnObserver.onSuccess(SingleSubscribeOn.java:68)
       at io.reactivex.internal.operators.single.SingleCreate$Emitter.onSuccess(SingleCreate.java:67)
       at com.androidcore.realm.AbstractDatabaseManager$getRealmEmitter$1.subscribe(AbstractDatabaseManager.java:161)
       at io.reactivex.internal.operators.single.SingleCreate.subscribeActual(SingleCreate.java:39)
       at io.reactivex.Single.subscribe(Single.java:3666)
       at io.reactivex.internal.operators.single.SingleSubscribeOn$SubscribeOnObserver.run(SingleSubscribeOn.java:89)
       at io.reactivex.android.schedulers.HandlerScheduler$ScheduledRunnable.run(HandlerScheduler.java:124)
       at android.os.Handler.handleCallback(Handler.java:739)
       at android.os.Handler.dispatchMessage(Handler.java:95)
       at android.os.Looper.loop(Looper.java:145)
       at android.os.HandlerThread.run(HandlerThread.java:61)

Steps & Code to Reproduce

Version of Realm and tooling

Realm version(s): 7.0.0
Realm Sync feature enabled: No
Android Studio version: 4.0.0
Android Build Tools version: 29.0.3
Gradle version: 6.5

[RealmBot]

➤ Brian Munkholm commented:

[~finn.schiermer-andersen] Have you seen this before?

[GaelCO]

No.
The method SalarieFactory.deleteSalarie exist in my project since a long time, but it was not called as much.
Before, it was called only when the user ask a log out. Whenth user log out, we remove all "salaries" in application.
Now, it's called at each launch if has obsolete "salarié" object.
The other changes is the upgrade to realm 7.0.0.

[bmunkholm]

@GaelCO How many is "a lot of users"? How big a % of users is it?
Can you possibly try to create tests for this yourself so you can reproduce what a lot of users seems to hit?

[bmunkholm]

It's most likely that you are trying to delete the same object twice. Perhaps check those places to guard against double deletion.

[GaelCO]

About 30% of users encounter this issue. This users could have a lot elements to remove.
I don't think I try to delete a same object twice. Moreover, I get the object before delete if.

`
private fun removeDocument(bgRealm: Realm, documentId: Long) {
get(bgRealm, documentId)?.deleteFromRealm()
}

    internal operator fun get(bgRealm: Realm, pkDocument: Long): Document? {
        return bgRealm.where(Document::class.java).equalTo(Document.Attributes.ID, pkDocument).findFirst()
    }`

[bmunkholm]

And you are 100% sure removeDocument() is the only place you delete?
That seems strange...
It would be awesome if you in any way could reproduce this in testing as we haven't seen others have this issue I believe.

[pstens]

Same as in #6994

[ChrisKruegerDev]

I have a similar issue with more than 2000 crashes in my app :(

Fatal Exception: io.realm.exceptions.RealmError: Unrecoverable error. Key not found in /Users/cm/Realm/realm-java-release/realm/realm-library/src/main/cpp/io_realm_internal_OsResults.cpp line 475 at io.realm.internal.OsResults.nativeEvaluateQueryIfNeeded(OsResults.java) at io.realm.internal.OsResults.load(OsResults.java:664) at io.realm.RealmResults.load(RealmResults.java:142) at io.realm.RealmQuery.createRealmResults(RealmQuery.java:2397) at io.realm.RealmQuery.findAll(RealmQuery.java:1778) at com.moviebase.ui.home.shard.RealmListValuesHelper.get(RealmListValuesHelper.java:61)

[bmunkholm]

@pstens @ChrisBLN Do you have any way to reproduce this? Or any hints as to what is happening before this occurs?

[pstens]

This occurred only after upgrading from 6.1.0 to 7.0.1.

The crash itself happened after downloading a bunch of entities from a remote API and then using Realm’s insertOrUpdate().
Unfortunately, I don’t have more information myself :/

[ChrisKruegerDev]

This occurred only after upgrading from 6.1.0 to 7.0.1.

The crash itself happened after downloading a bunch of entities from a remote API and then using Realm’s insertOrUpdate().
Unfortunately, I don’t have more information myself :/

Yes, I have a similar case.
The crashes happen in the main thread when the user opens the app. Users mentioned that it crashes when they sync some data in a background task (could be a few hundred items), which starts in a separate thread at the launch time.

[cmelchior]

I tried to reproduce this without being able. @GaelCO or @ChrisBLN if any of you have access to a Realm or a demo project that are able to reproduce this, it would be very helpful.

[ChrisKruegerDev]

I am in a huge trouble with this issue :( The app is running for many users, but when they delete the app data then the next startup it crashes. Mostly the mentioned issue, but also sometimes this one #6995.

The app uses Realm without big issues since 2016. Is it possible to downgrade to Realm 6?

[cmelchior]

Are they deleting the app data when the app is running? Because in that case, I wouldn't be surprised there are weird crashes. But if you close the app, delete the app data and reopen the data I would expect things to work.

I guess the big question is that how your code can get the above stacktrace if the Realm is empty? 🤔

In terms of downgrading, then you cannot downgrade an upgraded file, but if you delete the file, then it should be possible to downgrade. Not sure if that is an option for you?

[ChrisKruegerDev]

When I run the app it is working perfectly.

The users are deleting the app data when the app is closed and I think the app crashes when syncing a lot of data in another thread at the launch time. As you already said, it will create a completely new default Realm file and this shouldn't be the problem. I will test the issue a bit more.

Hm, I think downgrading would end up in more problems and it is just running away from the real issue...

Thanks for your help :)

[ChrisKruegerDev]

@cmelchior What is the meaning of "Unrecoverable error. Key not found"?

[bmunkholm]

Each Object has a Key in the database. You get this error when you try to delete an Object that doesn't exist. But we can't rule out something fishy is going on, so a repro-case would be awesome and enable to either explain what you are doing wrong or determine there is indeed a bug.

[jd565]

We are seeing a similar issue (we created #6994), but when trying to read an item from the Realm.
We have a RealmProfileAction, which contains a RealmMember, and are getting the key not found error when trying to get the nested RealmMember.
In our codebase, we never delete the RealmMember so either something is happening inside Realm to delete the entry, or Realm is calculating this key differently at different times.

We are also seeing this error when trying to insertOrUpdate a RealmMember, as well as an error that a key already exists when running insertOrUpdate on a RealmMember.

We have not been able to reproduce ourselves but was effecting about 1% of our users over a wide range of devices before we downgraded realm back to 6.1.0, and this is now stopping us from upgrading until this issue can be fixed.

[ChrisKruegerDev]

Seems that I could find the issue. The app has executed around 2200 realm transactions.

The transaction was similar like:

realm.executeTransaction {
 val managedMovie = realm.copyToRealmOrUpdate(movie)
  realm.where(RealmMediaWrapper::class.java).findAll().forEach {
    it.setContent(managedMovie)
  }
}

[rorbech]

I still have not been able to reproduce by iterating a lot of creation/updates with cross-reference updates.

Are there any patterns on the affected devices, e.g. 32/64bit, specific brands, etc.?

[GaelCO]

Hi,

I send a file with this behavior. This file contains fake data.
I generate with the test project I share previously.
Besides, I am surprised that you do not reproduce the problem with it.

realm.zip

[GaelCO]

Out of curiosity, I tried a "deleteAll". I have got an error too. It's not exactly the same (see the track).
After that, each deleteAll conclude to this error, but my other delete passed.
But, if I add new objects and clean it, I get the first error again. It looks like an index problem.

io.realm.exceptions.RealmError: Unrecoverable error. Key not found in /Users/cm/Realm/realm-java-release/realm/realm-library/src/main/cpp/io_realm_internal_Table.cpp line 190
        at io.realm.internal.Table.nativeClear(Native Method)
        at io.realm.internal.Table.clear(Table.java:262)
        at io.realm.BaseRealm.deleteAll(BaseRealm.java:741)
        at io.realm.Realm.deleteAll(Realm.java:145)
        at com.gco.realmtest.MainActivity$deleteAllCompletable$1.write(MainActivity.kt:114)
        at com.gco.realmtest.realm.DatabaseWriteHelper$getWriterCompletable$1$1.execute(DatabaseWriteHelper.kt:39)
        at io.realm.Realm.executeTransaction(Realm.java:1489)
        at com.gco.realmtest.realm.DatabaseWriteHelper$getWriterCompletable$1.subscribe(DatabaseWriteHelper.kt:38)
        at io.reactivex.internal.operators.completable.CompletableCreate.subscribeActual(CompletableCreate.java:39)
        at io.reactivex.Completable.subscribe(Completable.java:2309)
        at io.reactivex.internal.operators.single.SingleFlatMapCompletable$FlatMapCompletableObserver.onSuccess(SingleFlatMapCompletable.java:91)
        at io.reactivex.internal.operators.single.SingleSubscribeOn$SubscribeOnObserver.onSuccess(SingleSubscribeOn.java:68)
        at io.reactivex.internal.operators.single.SingleCreate$Emitter.onSuccess(SingleCreate.java:67)
        at com.gco.realmtest.realm.AbstractDatabaseManager$getRealmEmitter$1.subscribe(AbstractDatabaseManager.kt:73)
        at io.reactivex.internal.operators.single.SingleCreate.subscribeActual(SingleCreate.java:39)
        at io.reactivex.Single.subscribe(Single.java:3666)
        at io.reactivex.internal.operators.single.SingleSubscribeOn$SubscribeOnObserver.run(SingleSubscribeOn.java:89)
        at io.reactivex.android.schedulers.HandlerScheduler$ScheduledRunnable.run(HandlerScheduler.java:124)
        at android.os.Handler.handleCallback(Handler.java:938)
        at android.os.Handler.dispatchMessage(Handler.java:99)
        at android.os.Looper.loop(Looper.java:223)
        at android.os.HandlerThread.run(HandlerThread.java:67)

[jedelbo]

I now have a good lead on this.

[afaucogney]

Same bug here. Is the realm-core fix already affect a Java release ? If not, what/when will be this released out (Sorry I do not have deep knowledge of or release process) ?

Thanks

[clementetb]

@afaucogney The fix hasn't been released yet on realm-java. It will be available in the next release.

[afaucogney]

Any schedule for the next release ? At least an estimation would be nice.

[rorbech]

@GaelCO @ChrisBLN @afaucogney Sorry for not updating the issue. The fix is included from 10.1.0 from November 23th. Any feedback on the fix is most appreciated.

[afaucogney]

Thanks @rorbech. I had no plan to just upgrade to 10 just because of the issue ?
Is there any plan to also fix in 7.x ?
BTW, is there any issue when upgrading to 10x, I always experience some difficulties when upgrading Realm version, tbh !

[Jeff11]

I can't find it in the changelog of 10.1.0, @rorbech, @clementetb.

Has someone already updated to 10.1.x and noticed an improvement?

[rorbech]

@Jeff11 Sorry. The core fixes seems to have slipped from the change log. The fix (realm/realm-core#4051) was included by the upgrade to Core 10.1.3.

[jedelbo]

Assume this has been solved

[GaelCO]

That's not fixed the data corruption. With version 10.2.0, a crash occurs as soon as I touch the database.
2020-12-14 17:22:01.365 5818-7380/com.gco.realmtest E/REALM_JNI: jni: ThrowingException 5, No such object in /tmp/realm-java/realm/realm-library/src/main/cpp/io_realm_internal_OsSharedRealm.cpp line 106, .
2020-12-14 17:22:01.365 5818-7380/com.gco.realmtest E/REALM_JNI: Exception has been thrown: Unrecoverable error. No such object in /tmp/realm-java/realm/realm-library/src/main/cpp/io_realm_internal_OsSharedRealm.cpp line 106
2020-12-14 17:22:01.367 5818-7380/com.gco.realmtest E/AndroidRuntime: FATAL EXCEPTION: RealmThread_write
Process: com.gco.realmtest, PID: 5818
io.realm.exceptions.RealmError: Unrecoverable error. No such object in /tmp/realm-java/realm/realm-library/src/main/cpp/io_realm_internal_OsSharedRealm.cpp line 106
at io.realm.internal.OsSharedRealm.nativeGetSharedRealm(Native Method)
at io.realm.internal.OsSharedRealm.(OsSharedRealm.java:173)
at io.realm.internal.OsSharedRealm.getInstance(OsSharedRealm.java:249)
at io.realm.BaseRealm.(BaseRealm.java:138)
at io.realm.BaseRealm.(BaseRealm.java:105)
at io.realm.Realm.(Realm.java:159)
at io.realm.Realm.createInstance(Realm.java:495)
at io.realm.RealmCache.createInstance(RealmCache.java:481)
at io.realm.RealmCache.doCreateRealmOrGetFromCache(RealmCache.java:448)
at io.realm.RealmCache.createRealmOrGetFromCache(RealmCache.java:412)
at io.realm.Realm.getInstance(Realm.java:424)
at com.gco.realmtest.realm.RealmThread.onLooperPrepared(RealmThread.kt:31)
at android.os.HandlerThread.run(HandlerThread.java:66)

[edualonso]

@GaelCO sorry to hear about this. When you say a crash occurs as soon as I touch the database is it when you obtain a Realm instance or when you run a query or a transaction? I will pass the info to the core team as soon as I hear from you.

[edualonso]

According to the Core team the original problem reported in this ticket should be fixed in realm/realm-core#4051 and was included in the latest Realm 10.1.3 release.

However, once a Realm becomes corrupt, which could happen when upgrading from 6.x.x to 10.0.x (before the fix for the original "key not found" issue was released), it will continue to be so and, unfortunately, there is nothing we can do about it - see realm/realm-core#4051 (comment).

With that in mind, it would be interesting to know if there are still instances of the original "key not found" crash after updating to version 10.1.3.

[GaelCO]

It's when a run a query. (in my test project, add or delete items)

[ptsiogas]

It started to happen in a project that I am working when we updated realm gradle plugin from 5.9.0 to 7.0.0.
Firebase is reporting that this crash is suddenly affecting the 2-3% of our user-base. We have tried to replicate it with many different ways but it was not possible.

• We have unencrypted realm and the crash occurs on app-startup.
• Most of the crashing devices have >1GB available memory when crashing

Fatal Exception: io.realm.exceptions.RealmError: Unrecoverable error. Key not found in /Users/cm/Realm/realm-java-release/realm/realm-library/src/main/cpp/io_realm_internal_OsSharedRealm.cpp line 107
       at io.realm.internal.OsSharedRealm.nativeGetSharedRealm(OsSharedRealm.java)
       at io.realm.internal.OsSharedRealm.<init>(OsSharedRealm.java:175)
       at io.realm.internal.OsSharedRealm.getInstance(OsSharedRealm.java:251)
       at io.realm.BaseRealm.<init>(BaseRealm.java:137)
       at io.realm.BaseRealm.<init>(BaseRealm.java:104)
       at io.realm.Realm.<init>(Realm.java:163)
       at io.realm.Realm.createInstance(Realm.java:499)
       at io.realm.RealmCache.createInstance(RealmCache.java:507)
       at io.realm.RealmCache.doCreateRealmOrGetFromCache(RealmCache.java:473)
       at io.realm.RealmCache.createRealmOrGetFromCache(RealmCache.java:414)
       at io.realm.Realm.getDefaultInstance(Realm.java:407)

Is there any fix or workaround for the issue?

[clementetb]

@ptsiogas This issue should have been fixed in version 10.1.3, could you try to reproduce the issue on it?

[jedelbo]

I take the liberty to close this issue again as the original problem on 32-bit devices has been solved. I anyone encounters new issues, please create new tickets for them.