Database corruption after SEGV_MAPERR #4128
Comments
|
Hi @bios-seiji Thank you for a very detailed bug report 🎉 . If you can send the project to help@realm.io we would be extremely grateful. |
|
Can you try to change the following to see if it helps stabilizing the issue? Thread mWriteThread = new Thread(new Runnable() {
@Override
public void run() {
final AtomicInteger i = new AtomicInteger(0);
while(mRunThreads) {
try(Realm realm = Realm.getInstance(REALM_CONFIG)) {
realm.executeTransaction(new Realm.Transaction() {
@Override
public void execute(Realm realm) {
Contact contact = realm.where(Contact.class).equalTo("id", CONTACT_ID).findFirst();
contact.setUsername("User" + i.incrementAndGet());
}
});
}
}
}
});
Thread mReadThread = new Thread(new Runnable() {
@Override
public void run() {
while(mRunThreads) {
try(Realm realm = Realm.getInstance(REALM_CONFIG)) {
while(mRunThreads) {
try(Realm realm = Realm.getInstance(REALM_CONFIG)) {
realm.where(Contact.class).equalTo("id", CONTACT_ID).findAllSorted("username");
realm.copyFromRealm(realm.where(Contact.class).equalTo("id", CONTACT_ID).findAllSorted("username"), 0);
}
}
}
}
}
});to Thread mWriteThread = new Thread(new Runnable() {
@Override
public void run() {
final AtomicInteger i = new AtomicInteger(0);
try(Realm realm = Realm.getInstance(REALM_CONFIG)) {
while(mRunThreads) {
try(Realm realm = Realm.getInstance(REALM_CONFIG)) {
realm.executeTransaction(new Realm.Transaction() {
@Override
public void execute(Realm realm) {
Contact contact = realm.where(Contact.class).equalTo("id", CONTACT_ID).findFirst();
contact.setUsername("User" + i.incrementAndGet());
}
});
}
}
}
}
});
Thread mReadThread = new Thread(new Runnable() {
@Override
public void run() {
try(Realm realm = Realm.getInstance(REALM_CONFIG)) {
while(mRunThreads) {
try(Realm realm = Realm.getInstance(REALM_CONFIG)) {
realm.where(Contact.class).equalTo("id", CONTACT_ID).findAllSorted("username");
realm.copyFromRealm(realm.where(Contact.class).equalTo("id", CONTACT_ID).findAllSorted("username"), 0);
}
}
}
}
}); |
|
@Zhuinden apologies, that was a typo in my C&P (I corrected the original post). I have already been conducting tests with the change you suggest. |
|
Here's the code we've been using to test load (and generate crashes): |
|
@bios-seiji Thanks! |
|
@kneth were you able to reproduce the crashes with the project I provided? Do you have any suggestions for further debugging or mitigation? Once a user has a corrupted database, is there any way we can detect and triage? Currently, the application will try to run and crash unexpectedly and uncontrollably. At the least we should be able to detect corruption on boot and provide steps for the user to get re-setup. Thanks |
|
@bios-seiji Sorry for not getting back earlier. The code described above is similar to the code found in #4114. Which isn't surprising as #4114 was written with realm/realm-core#2383 in mind. But the reason for the crash in #4114 might come from the fact that some threads are starved (22 threads and only single core in my emulator) so a thread might hold an old version of database and the device runs out of physical memory (bus error indicates that). Do you see the crash if you disable encryption? By the way, the test app - does it require API 23 to fail? How many iterations do it typically take to crash? @ironage You might wish to take a look at the test app while debugging realm/realm-core#2383. |
|
@kneth I can reliably reproduce these crashes on 4 and 8 core devices running as few as 3 or 4 threads (see example cases https://github.com/bios-seiji/realm-crash/blob/master/app/src/main/java/io/binarysolutions/realmtest/MainActivity.java#L29). The devices report ample physical memory available (dumpsys meminfo), and running the app with largeHeap enabled does not seem to help. I also dumped the heap an looked at it in MAP and did not see any obvious leaks. What are you using to monitor when "device runs out of physical memory"? This was my initial thought also, but I have not seen it. In our production app, I disabled encryption and was able to reproduce the issue with much more effort. I have not done extensive testing however, since encryption is necessary for us. I recompiled to Android 5.1 (API 22) and was able to reproduce the issues. I did not get immediate crash described in my test cases, but it crashed between 1-10k write iterations reliably. |
|
@bios-seiji The investigations have so far lead to realm/realm-core#2426. We need to test more. |
|
@kneth thats great. Do you have a testing build that I can run with our production tests to see if I can still trigger corruption? |
|
@bios-seiji Currently I am getting ready to test with your test app (no reason to involve you before I have done that). |
|
I am currently running your app in a x86 emulator. So far, thread 1 is at 34k, thread 2 is at 13k and duration at 1102k. @bios-seiji If you wish to try my custom build, please send an email to help@realm.io. |
|
@bios-seiji Your detailed bug report made it possible for us to reproduce a bug in Realm Core. It has now been fixed (see realm/realm-core#2465), and we will release a fixed very soon. I am closing the issue, and we are very thankful for your original report. |
Hello,
We're experiencing high incidents of SIGSEGV SEGV_MAPERR crashes on recent versions of Realm. We can reliably reproduce these crashes by opening realms on multiple threads and conducting operations. Crashing multiple times (1-5) will usually cause the database to corrupt. Most of our tests have been with encrypted realms, however, I was able to reproduce the same results using an unencrypted realm with much effort.
Observations
Results
This is a small set of crashes from production. I have hundreds, but they're much of the same.
Steps & Code to Reproduce
Encryption: On. Minimum 1 contact, 1 message, with body.length ~= 1000.
At least 1 write thread, and 1 other thread. Global instance N=2 will eventually crash, N=3+ will crash faster.
This code will often produce UTF-16 crash for us as well. We can provide the project if necessary, but this is the relevant code.
Mitigations
Reducing global realm instances to N<=2 and reducing as much load on Realm as possible decreases the chances of crash and corruption.
Version of Realm and tooling
Realm version(s): 2.2.1, 2.2.2, 2.3.0
Realm sync feature enabled: no
Encryption: yes
Android Studio version: 2.2.2
Which Android version and device: Android 6.0.1 (CM13) / CAF 6.0.1, OnePlus One / OnePlus X
Next steps
This appears to be similar to issue reported here: realm/realm-core#2383
How can we further debug and fix these issues?
Do you have any way to detect and/or recover from corruption? ie open realm in RO and verify integrity.
The text was updated successfully, but these errors were encountered: