Null pointer crash in create_object_with_primary_key every time for certain DB

[tfe]

How frequently does the bug occur?

All the time

Description

This started happening after updating from Realm 10.7.5 to 10.21.1 (and does not happen if I revert back to 10.7.5).

The database regularly gets into a state where any object creation will crash, every time. I've reduced the app down to the most minimal case, doing nothing upon launch but the following, and it still does it.

// Verified no Course objects in DB via `realm.objects(_ type:)`
let course = Course()
course.id = "123abc" // primary key, tried a couple values in case an old one was poisoned somehow
MyDataManager.shared.save(course, in: .courses) // gets the realm, calls `realm.add(course)`

If I delete the app bundle and re-install, it might work a couple times but pretty quickly gets into whatever corrupt state is causing this, after which it's unusable until it's deleted and recreated.

This tells me there is something wrong with the database files, so I've preserved those and would be happy to provide those privately for analysis to solve this bug.

In case it's relevant, the way we use this database is that it's created by copying a seed database containing large object tree (the "course"). This part is working fine. The problem starts happening when we do an operation to update the data, which involves clearing the database (deleting all objects) before downloading, decoding, and then adding the new object tree to the realm.

This makes me suspect something in the deletion as the root cause, but I'm hoping inspection of my Realm data files will make it obvious what's wrong with them.

Stacktrace & log output

* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
    frame #0: 0x00000001127db059 Realm`realm::Table::create_object_with_primary_key(realm::Mixed const&, std::__1::vector<realm::FieldValue, std::__1::allocator<realm::FieldValue> >&&, bool*) + 233
  * frame #1: 0x000000011188b5a4 Realm`realm::Table::create_object_with_primary_key(this=0x000061b000031780, primary_key=0x00007ffee6d2aaf0, did_create=0x00007ffee6d2a890) at table.hpp:245:16
    frame #2: 0x00000001117a75a4 Realm`realm::Object realm::Object::create<objc_object* __strong, RLMAccessorContext>(ctx=0x00007ffee6d2c050, realm=std::__1::shared_ptr<realm::Realm>::element_type @ 0x00006170000ad120 strong=1 weak=7, object_schema=0x000061d000118d00, value=0x00006130000cf2c0, policy=(create = true, copy = false, update = false, diff = false), current_obj=(value = -1), out_row=0x00006130000cf2c8) at object_accessor.hpp:298:26
    frame #3: 0x00000001117a497c Realm`RLMAccessorContext::createObject(this=0x00007ffee6d2c050, value=0x00006130000cf2c0, policy=(create = true, copy = false, update = false, diff = false), forceCreate=false, existingKey=(value = -1)) at RLMAccessor.mm:1093:9
    frame #4: 0x0000000111c75c6a Realm`RLMAddObjectToRealm(object=0x00006130000cf2c0, realm=0x000060f000042ee0, updatePolicy=RLMUpdatePolicyError) at RLMObjectStore.mm:138:7
    frame #5: 0x000000010fa92d8e RealmSwift`Realm.add(object=0x00006130000cf2c0, update=error, self=RealmSwift.Realm @ 0x00007ffee6d2c350) at Realm.swift:432:9
    frame #6: 0x0000000109215fb7 MyApp`MyDataManager.save<T>(objects=1 value, db=courses, update=nil, replaceExisting=true, self=0x00006040000db010) at MyDataManager.swift:485:23
    frame #7: 0x0000000109215584 MyApp`MyDataManager.save<T>(object=0x00006130000cf2c0, db=courses, update=nil, replaceExisting=true, self=0x00006040000db010) at MyDataManager.swift:471:9
    frame #8: 0x0000000109207e4b MyApp`MyDataManager.fetchAndInstallCoursesData(self=0x00006040000db010) at MyDataManager.swift:225:14
    frame #9: 0x000000010948a6b0 MyApp`AppViewController.courseSelectionViewControllerReloadButtonTapped(sender=0x000061800008dc80, self=0x0000619000186f80) at AppViewController.swift:812:30
    frame #10: 0x000000010948b249 MyApp`protocol witness for CourseSelectionViewControllerDelegate.courseSelectionViewControllerReloadButtonTapped(_:) in conformance AppViewController at <compiler-generated>:0
    frame #11: 0x0000000109519692 MyApp`CourseSelectionViewController.reloadCoursesButtonTapped(sender=0x00006170000b0200, self=0x000061800008dc80) at CourseSelectionViewController.swift:52:19
    frame #12: 0x000000010951979a MyApp`@objc CourseSelectionViewController.reloadCoursesButtonTapped(_:) at <compiler-generated>:0
    frame #13: 0x000000012dd5326d UIKitCore`-[UIApplication sendAction:to:from:forEvent:] + 83
    frame #14: 0x000000012d5e96e1 UIKitCore`-[UIControl sendAction:to:forEvent:] + 110
    frame #15: 0x000000012d5e9ac3 UIKitCore`-[UIControl _sendActionsForEvents:withEvent:] + 332
    frame #16: 0x000000012d5e5ffa UIKitCore`-[UIButton _sendActionsForEvents:withEvent:] + 148
    frame #17: 0x000000012d5e8374 UIKitCore`-[UIControl touchesEnded:withEvent:] + 488
    frame #18: 0x000000012dd93c9d UIKitCore`-[UIWindow _sendTouchesForEvent:] + 1287
    frame #19: 0x000000012dd95d19 UIKitCore`-[UIWindow sendEvent:] + 5289
    frame #20: 0x000000012dd6c5b6 UIKitCore`-[UIApplication sendEvent:] + 819
    frame #21: 0x000000010c996017 FLEX`__35+[FLEXKeyboardShortcutManager load]_block_invoke.104(.block_descriptor=0x0000604000071f50, slf=0x000061400000cc40, event=0x000061000002fe40) at FLEXKeyboardShortcutManager.m:177:13
    frame #22: 0x000000012de024ef UIKitCore`__dispatchPreprocessedEventFromEventQueue + 8683
    frame #23: 0x000000012de04c75 UIKitCore`__processEventQueue + 8579
    frame #24: 0x000000012ddfb427 UIKitCore`__eventFetcherSourceCallback + 240
    frame #25: 0x000000011939fe15 CoreFoundation`__CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 17
    frame #26: 0x000000011939fd0d CoreFoundation`__CFRunLoopDoSource0 + 180
    frame #27: 0x000000011939f1e2 CoreFoundation`__CFRunLoopDoSources0 + 242
    frame #28: 0x0000000119399941 CoreFoundation`__CFRunLoopRun + 875
    frame #29: 0x00000001193990f3 CoreFoundation`CFRunLoopRunSpecific + 567
    frame #30: 0x000000011e8a3cd3 GraphicsServices`GSEventRunModal + 139
    frame #31: 0x000000012dd4cf42 UIKitCore`-[UIApplication _run] + 928
    frame #32: 0x000000012dd51b5e UIKitCore`UIApplicationMain + 101
    frame #33: 0x00000001094f74df MyApp`main at AppDelegate.swift:19:13
    frame #34: 0x000000010a6b7ee9 dyld_sim`start_sim + 10

Can you reproduce the bug?

Yes, always

Reproduction Steps

As I mentioned, it seems to be something with the specific database, so the most relevant thing for reproducing is those files which I can provide privately.

Version

10.21.1

What SDK flavour are you using?

Local Database only

Are you using encryption?

Yes, using encryption

Platform OS and version(s)

iOS 14 and 15

Build environment

Version 13.2.1 (13C100)
Dependency manager and version: cocoa pods

[dianaafanador3]

Hi @tfe in order to reproduce this issue, would you mind sharing your MyDataManager class and how do you create your realm instance as well, also the code where you clear your database and adds the new object, lastly your objects schema will be also very helpful.

[tfe]

Hi @dianaafanador3, thanks! I can provide those. I put together a gist with the relevant files, with less relevant parts trimmed for length and readability. I still included the functions responsible for initial creation of the realm by copying the seed data realm.

Here are some details to help you navigate the files in the gist:

how do you create your realm instance

This will be in DataManager.swift, look in the getRealm and openRealm functions.

the code where you clear your database and adds the new object

This will be in DataManager.swift as well, look in the fetchAndInstallCoursesData and clearData functions, as well as save.

lastly your objects schema will be also very helpful

I've included Course.swift in the gist, stripped down to just show the properties and how we're mapping them from the server response using ObjectMapper.

[tfe]

@dianaafanador3 Just checking in on this. Can I provide anything else to help?

[ejm01]

What happens in overwriteFile(with: Realm)? Does it copy data from one realm to another existing? Or does it create a new file at the same path?

The curious thing about the .courses realm is that seems like it can exist for a very long time, because a lot of operations are copying, clearing data, etc. through transactions. And doesn't appear to use a new file.
Combine this with the fact that 10.7.5 is working. I'm wondering if something didn't get properly upgraded in the realm file and it's really old?

Maybe using .writeCopy instead of transactions will help. I'm not certain. But just an idea.

[tfe]

@ericjordanmossman I'll paste the code for that function (and related) below. I agree that it seems like something is going wrong with the file, however it's not because it's old. As I mentioned, it happens even on fresh installs:

If I delete the app bundle and re-install, it might work a couple times but pretty quickly gets into whatever corrupt state is causing this, after which it's unusable until it's deleted and recreated.

If I provided the Realm file that's exhibiting this behavior, would you or your team be able to analyze them to see what's going on to trigger the create_object_with_primary_key crash that I'm seeing?

Here are the relevant parts of the struct containing overwriteFile(with: Realm):

struct RealmConfigData {
    let fileName: String

    var dataFileName: String {
        return "\(fileName)\(EnvironmentManager.shared.dataStoreSuffix).realm"
    }

    var fileURL: URL {
        return Realm.Configuration.defaultConfiguration.fileURL!
            .deletingLastPathComponent()
            .appendingPathComponent(dataFileName)
    }

    func overwriteFile(with realm: Realm) throws {
        // first remove existing if present
        do {
            try FileManager.default.removeItem(at: fileURL)
        } catch CocoaError.fileNoSuchFile {
            // This is fine; the file isn't always there. Continue.
        }

        try realm.writeCopy(toFile: fileURL, encryptionKey: Self.encryptionKeyData(from: encryptionKey))
    }
}

[dianaafanador3]

Hi @tfe If you can send us a copy of the corrupted realm we may diagnose the issue. Please send it to diana.perez@mongodb.com

[tfe]

@dianaafanador3 Thanks so much! Sent it to you via email.

[dianaafanador3]

Hi @tfe we were not able to decrypt your realm realm file, which could mean that the file is completely corrupted. Have you tried downgrading to 10.20.2 which was suggested on the other issue. We are trying to identify the issue and fix this as soon as possible.

[leemaguire]

@tfe can you show us what the implementation of Self.encryptionKeyData(from:) looks like?

[leemaguire]

When this crash occurs, are you using Realm on any other thread? Are you able to provide more stack traces?

[tfe]

@dianaafanador3 It opens without complaint in my app (only crashing when trying to create an object), so I'm not sure what to say there.

@leemaguire I don't have any more stack traces, sorry. This never got shipped to more than one or two users before we saw the issue, so I don't have any others. I'll paste Self.encryptionKeyData(from:) below.

Regarding threads: yes, it's possible... I mentioned the process where we update the data in this realm (by deleting most objects, downloading, decoding, and then adding a new object tree). After that completes, we kick off a task on a background thread to clean up orphaned objects.

static func encryptionKeyData(from key: String?) -> Data? {
    guard let key = key else { return nil }
    return key.data(using: String.Encoding.utf8, allowLossyConversion: false)
}

[stoneyMDB]

Moved to realm/realm-core#5397